Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines
Hi Lincoln, Thanks for driving this discussion, and I am so excited to see this topic being discussed in the Flink community! >From my point of view, instead of the work of unifying streaming and batch in >DataStream API [1], this FLIP actually could make users benefit from one >engine to rule batch & streaming. If we treat this FLIP as an open-source implementation of Snowflake's dynamic tables [2], we still lack an incremental refresh mode to make the ETL near real-time with a much cheaper computation cost. However, I think this could be done under the current design by introducing another refresh mode in the future. Although the extra work of incremental view maintenance would be much larger. For the FLIP itself, I have several questions below: 1. It seems this FLIP does not consider the lag of refreshes across ETL layers from ODS ---> DWD ---> APP [3]. We currently only consider the scheduler interval, which means we cannot use lag to automatically schedule the upfront micro-batch jobs to do the work. 2. To support the automagical refreshes, we should consider the lineage in the catalog or somewhere else. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-134%3A+Batch+execution+for+the+DataStream+API [2] https://docs.snowflake.com/en/user-guide/dynamic-tables-about [3] https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh Best Yun Tang From: Lincoln Lee Sent: Thursday, March 14, 2024 14:35 To: dev@flink.apache.org Subject: Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines Hi Jing, Thanks for your attention to this flip! I'll try to answer the following questions. > 1. How to define query of dynamic table? > Use flink sql or introducing new syntax? > If use flink sql, how to handle the difference in SQL between streaming and > batch processing? > For example, a query including window aggregate based on processing time? > or a query including global order by? Similar to `CREATE TABLE AS query`, here the `query` also uses Flink sql and doesn't introduce a totally new syntax. We will not change the status respect to the difference in functionality of flink sql itself on streaming and batch, for example, the proctime window agg on streaming and global sort on batch that you mentioned, in fact, do not work properly in the other mode, so when the user modifies the refresh mode of a dynamic table that is not supported, we will throw an exception. > 2. Whether modify the query of dynamic table is allowed? > Or we could only refresh a dynamic table based on the initial query? Yes, in the current design, the query definition of the dynamic table is not allowed to be modified, and you can only refresh the data based on the initial definition. > 3. How to use dynamic table? > The dynamic table seems to be similar to the materialized view. Will we do > something like materialized view rewriting during the optimization? It's true that dynamic table and materialized view are similar in some ways, but as Ron explains there are differences. In terms of optimization, automated materialization discovery similar to that supported by calcite is also a potential possibility, perhaps with the addition of automated rewriting in the future. Best, Lincoln Lee Ron liu 于2024年3月14日周四 14:01写道： > Hi, Timo > > Sorry for later response, thanks for your feedback. > Regarding your questions: > > > Flink has introduced the concept of Dynamic Tables many years ago. How > > does the term "Dynamic Table" fit into Flink's regular tables and also > > how does it relate to Table API? > > > > I fear that adding the DYNAMIC TABLE keyword could cause confusion for > > users, because a term for regular CREATE TABLE (that can be "kind of > > dynamic" as well and is backed by a changelog) is then missing. Also > > given that we call our connectors for those tables, DynamicTableSource > > and DynamicTableSink. > > > > In general, I find it contradicting that a TABLE can be "paused" or > > "resumed". From an English language perspective, this does sound > > incorrect. In my opinion (without much research yet), a continuous > > updating trigger should rather be modelled as a CREATE MATERIALIZED VIEW > > (which users are familiar with?) or a new concept such as a CREATE TASK > > (that can be paused and resumed?). > > > 1. > In the current concept[1], it actually includes: Dynamic Tables & > Continuous Query. Dynamic Table is just an abstract > logical concept > , which in its physical form represents either a table or a changelog > stream. It requires the combination with Continuous Query to achieve > dynamic updates of the target table similar to a database’s > Materialized View. > We hope to upgrade the Dynamic Table to a real entity that users can > operate, which combines the logical concepts of Dynamic Tables + > Continuous Query. By integrating the definition of tables and queries, > it can achieve functions
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congrats and thanks all for the efforts! Best Regards, Yu On Tue, 19 Mar 2024 at 11:51, gongzhongqiang wrote: > > Congrats! Thanks to everyone involved! > > Best, > Zhongqiang Gong > > Lincoln Lee 于2024年3月18日周一 16:27写道： >> >> The Apache Flink community is very happy to announce the release of Apache >> Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 series. >> >> Apache Flink® is an open-source stream processing framework for >> distributed, high-performing, always-available, and accurate data streaming >> applications. >> >> The release is available for download at: >> https://flink.apache.org/downloads.html >> >> Please check out the release blog post for an overview of the improvements >> for this bugfix release: >> https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ >> >> The full release notes are available in Jira: >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 >> >> We would like to thank all contributors of the Apache Flink community who >> made this release possible! >> >> >> Best, >> Yun, Jing, Martijn and Lincoln
Support minibatch for TopNFunction
Hello Flink Community, The same problem with record amplification as described in FLIP-415: Introduce a new join operator to support minibatch[1] exists for most of implementations of AbstractTopNFunction. Especially when the rank is provided to output. For example, when calculating Top100 with rank output, every input record might produce 100 -U records and 100 +U records. According to my POC (which is similar to FLIP-415) the record amplification could be significantly reduced by using input or output buffer. What do you think if we implement such optimization for TopNFunctions? [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-415 %3A+Introduce+a+new+join+operator+to+support+minibatch -- Best regards, Roman Boyko e.: ro.v.bo...@gmail.com m.: +79059592443 telegram: @rboyko
Re: [DISCUSS] FLIP-423 ～FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0
Thanks for the quick response. Sounds good to me. Best, Xintong On Tue, Mar 19, 2024 at 1:03 PM Zakelly Lan wrote: > Hi Xintong, > > Thanks for sharing your thoughts! > > 1. Regarding Record-ordered and State-ordered of processElement. > > > > I understand that while State-ordered likely provides better performance, > > Record-ordered is sometimes required for correctness. The question is how > > should a user choose between these two modes? My concern is that such a > > decision may require users to have in-depth knowledge about the Flink > > internals, and may lead to correctness issues if State-ordered is chosen > > improperly. > > > > I'd suggest not to expose such a knob, at least in the first version. > That > > means always use Record-ordered for custom operators / UDFs, and keep > > State-ordered for internal usages (built-in operators) only. > > > > Indeed, users may not be able to choose the mode properly. I agree to keep > such options for internal use. > > > 2. Regarding Strictly-ordered and Out-of-order of Watermarks. > > > > I'm not entirely sure about Strictly-ordered being the default, or even > > being supported. From my understanding, a Watermark(T) only suggests that > > all records with event time before T has arrived, and it has nothing to > do > > with whether records with event time after T has arrived or not. From > that > > perspective, preventing certain records from arriving before a Watermark > is > > never supported. I also cannot come up with any use case where > > Strictly-ordered is necessary. This implies the same issue as 1): how > does > > the user choose between the two modes? > > > > I'd suggest not expose the knob to users and only support Out-of-order, > > until we see a concrete use case that Strictly-ordered is needed. > > > > The semantics of watermarks do not define the sequence between a watermark > and subsequent records. For the most part, this is inconsequential, except > it may affect some current users who have previously relied on the implicit > assumption of an ordered execution. I'd be fine with initially supporting > only out-of-order processing. We may consider exposing the > 'Strictly-ordered' mode once we encounter a concrete use case that > necessitates it. > > > My philosophies behind not exposing the two config options are: > > - There are already too many options in Flink that barely know how to use > > them. I think Flink should try as much as possible to decide its own > > behavior, rather than throwing all the decisions to the users. > > - It's much harder to take back knobs than to introduce them. Therefore, > > options should be introduced only if concrete use cases are identified. > > > > I agree to keep minimal configurable items especially for the MVP. Given > that we have the opportunity to refine the functionality before the > framework transitions from @Experimental to @PublicEvolving, it makes sense > to refrain from presenting user-facing options until we have ensured > their necessity. > > > Best, > Zakelly > > On Tue, Mar 19, 2024 at 12:06 PM Xintong Song > wrote: > > > Sorry for joining the discussion late. > > > > I have two questions about FLIP-425. > > > > 1. Regarding Record-ordered and State-ordered of processElement. > > > > I understand that while State-ordered likely provides better performance, > > Record-ordered is sometimes required for correctness. The question is how > > should a user choose between these two modes? My concern is that such a > > decision may require users to have in-depth knowledge about the Flink > > internals, and may lead to correctness issues if State-ordered is chosen > > improperly. > > > > I'd suggest not to expose such a knob, at least in the first version. > That > > means always use Record-ordered for custom operators / UDFs, and keep > > State-ordered for internal usages (built-in operators) only. > > > > 2. Regarding Strictly-ordered and Out-of-order of Watermarks. > > > > I'm not entirely sure about Strictly-ordered being the default, or even > > being supported. From my understanding, a Watermark(T) only suggests that > > all records with event time before T has arrived, and it has nothing to > do > > with whether records with event time after T has arrived or not. From > that > > perspective, preventing certain records from arriving before a Watermark > is > > never supported. I also cannot come up with any use case where > > Strictly-ordered is necessary. This implies the same issue as 1): how > does > > the user choose between the two modes? > > > > I'd suggest not expose the knob to users and only support Out-of-order, > > until we see a concrete use case that Strictly-ordered is needed. > > > > > > My philosophies behind not exposing the two config options are: > > - There are already too many options in Flink that barely know how to use > > them. I think Flink should try as much as possible to decide its own > > behavior, rather than throwing all the decisions to the users.
Re: [DISCUSS] FLIP-423 ～FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0
Hi Xintong, Thanks for sharing your thoughts! 1. Regarding Record-ordered and State-ordered of processElement. > > I understand that while State-ordered likely provides better performance, > Record-ordered is sometimes required for correctness. The question is how > should a user choose between these two modes? My concern is that such a > decision may require users to have in-depth knowledge about the Flink > internals, and may lead to correctness issues if State-ordered is chosen > improperly. > > I'd suggest not to expose such a knob, at least in the first version. That > means always use Record-ordered for custom operators / UDFs, and keep > State-ordered for internal usages (built-in operators) only. > Indeed, users may not be able to choose the mode properly. I agree to keep such options for internal use. 2. Regarding Strictly-ordered and Out-of-order of Watermarks. > > I'm not entirely sure about Strictly-ordered being the default, or even > being supported. From my understanding, a Watermark(T) only suggests that > all records with event time before T has arrived, and it has nothing to do > with whether records with event time after T has arrived or not. From that > perspective, preventing certain records from arriving before a Watermark is > never supported. I also cannot come up with any use case where > Strictly-ordered is necessary. This implies the same issue as 1): how does > the user choose between the two modes? > > I'd suggest not expose the knob to users and only support Out-of-order, > until we see a concrete use case that Strictly-ordered is needed. > The semantics of watermarks do not define the sequence between a watermark and subsequent records. For the most part, this is inconsequential, except it may affect some current users who have previously relied on the implicit assumption of an ordered execution. I'd be fine with initially supporting only out-of-order processing. We may consider exposing the 'Strictly-ordered' mode once we encounter a concrete use case that necessitates it. My philosophies behind not exposing the two config options are: > - There are already too many options in Flink that barely know how to use > them. I think Flink should try as much as possible to decide its own > behavior, rather than throwing all the decisions to the users. > - It's much harder to take back knobs than to introduce them. Therefore, > options should be introduced only if concrete use cases are identified. > I agree to keep minimal configurable items especially for the MVP. Given that we have the opportunity to refine the functionality before the framework transitions from @Experimental to @PublicEvolving, it makes sense to refrain from presenting user-facing options until we have ensured their necessity. Best, Zakelly On Tue, Mar 19, 2024 at 12:06 PM Xintong Song wrote: > Sorry for joining the discussion late. > > I have two questions about FLIP-425. > > 1. Regarding Record-ordered and State-ordered of processElement. > > I understand that while State-ordered likely provides better performance, > Record-ordered is sometimes required for correctness. The question is how > should a user choose between these two modes? My concern is that such a > decision may require users to have in-depth knowledge about the Flink > internals, and may lead to correctness issues if State-ordered is chosen > improperly. > > I'd suggest not to expose such a knob, at least in the first version. That > means always use Record-ordered for custom operators / UDFs, and keep > State-ordered for internal usages (built-in operators) only. > > 2. Regarding Strictly-ordered and Out-of-order of Watermarks. > > I'm not entirely sure about Strictly-ordered being the default, or even > being supported. From my understanding, a Watermark(T) only suggests that > all records with event time before T has arrived, and it has nothing to do > with whether records with event time after T has arrived or not. From that > perspective, preventing certain records from arriving before a Watermark is > never supported. I also cannot come up with any use case where > Strictly-ordered is necessary. This implies the same issue as 1): how does > the user choose between the two modes? > > I'd suggest not expose the knob to users and only support Out-of-order, > until we see a concrete use case that Strictly-ordered is needed. > > > My philosophies behind not exposing the two config options are: > - There are already too many options in Flink that barely know how to use > them. I think Flink should try as much as possible to decide its own > behavior, rather than throwing all the decisions to the users. > - It's much harder to take back knobs than to introduce them. Therefore, > options should be introduced only if concrete use cases are identified. > > WDYT? > > Best, > > Xintong > > > > On Fri, Mar 8, 2024 at 2:45 AM Jing Ge wrote: > > > +1 for Gyula's suggestion. I just finished FLIP-423 which introduced the > > intention of
[jira] [Created] (FLINK-34725) Dockerfiles for release publishing has incorrect config.yaml path
lincoln lee created FLINK-34725:
---
Summary: Dockerfiles for release publishing has incorrect
config.yaml path
Key: FLINK-34725
URL: https://issues.apache.org/jira/browse/FLINK-34725
Project: Flink
Issue Type: Bug
Components: flink-docker
Affects Versions: 1.19.0
Reporter: lincoln lee
Fix For: 1.19.0
An issue found when do docker image publishing, unexpected error msg:
{code}sed: can't read /config.yaml: No such file or directory\{code}
also found in flink-docker/master daily Publish SNAPSHOTs action:
https://github.com/apache/flink-docker/actions/runs/8210534289/job/22458150514#step:8:588
[https://github.com/apache/flink-docker/actions/runs/8210534289/job/22458150322#step:8:549]
This related to changes by https://issues.apache.org/jira/browse/FLINK-34205
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [DISCUSS] FLIP-423 ～FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0
Sorry for joining the discussion late. I have two questions about FLIP-425. 1. Regarding Record-ordered and State-ordered of processElement. I understand that while State-ordered likely provides better performance, Record-ordered is sometimes required for correctness. The question is how should a user choose between these two modes? My concern is that such a decision may require users to have in-depth knowledge about the Flink internals, and may lead to correctness issues if State-ordered is chosen improperly. I'd suggest not to expose such a knob, at least in the first version. That means always use Record-ordered for custom operators / UDFs, and keep State-ordered for internal usages (built-in operators) only. 2. Regarding Strictly-ordered and Out-of-order of Watermarks. I'm not entirely sure about Strictly-ordered being the default, or even being supported. From my understanding, a Watermark(T) only suggests that all records with event time before T has arrived, and it has nothing to do with whether records with event time after T has arrived or not. From that perspective, preventing certain records from arriving before a Watermark is never supported. I also cannot come up with any use case where Strictly-ordered is necessary. This implies the same issue as 1): how does the user choose between the two modes? I'd suggest not expose the knob to users and only support Out-of-order, until we see a concrete use case that Strictly-ordered is needed. My philosophies behind not exposing the two config options are: - There are already too many options in Flink that barely know how to use them. I think Flink should try as much as possible to decide its own behavior, rather than throwing all the decisions to the users. - It's much harder to take back knobs than to introduce them. Therefore, options should be introduced only if concrete use cases are identified. WDYT? Best, Xintong On Fri, Mar 8, 2024 at 2:45 AM Jing Ge wrote: > +1 for Gyula's suggestion. I just finished FLIP-423 which introduced the > intention of the big change and high level architecture. Great content btw! > The only public interface change for this FLIP is one new config to use > ForSt. It makes sense to have one dedicated discussion thread for each > concrete system design. > > @Zakelly The links in your mail do not work except the last one, because > the FLIP-xxx has been included into the url like > https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864FLIP-425. > > NIT fix: > > FLIP-424: https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864 > > FLIP-425: https://lists.apache.org/thread/wxn1j848fnfkqsnrs947wh1wmj8n8z0h > > FLIP-426: https://lists.apache.org/thread/bt931focfl9971cwq194trmf3pkdsxrf > > FLIP-427: https://lists.apache.org/thread/vktfzqvb7t4rltg7fdlsyd9sfdmrc4ft > > FLIP-428: https://lists.apache.org/thread/vr8f91p715ct4lop6b3nr0fh4z5p312b > > Best regards, > Jing > > > > > On Thu, Mar 7, 2024 at 10:14 AM Zakelly Lan wrote: > > > Hi everyone, > > > > Thank you all for a lively discussion here, and it is a good time to move > > forward to more detailed discussions. Thus we open several threads for > > sub-FLIPs: > > > > FLIP-424: > https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864 > > FLIP-425 > > < > https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864FLIP-425>: > > https://lists.apache.org/thread/wxn1j848fnfkqsnrs947wh1wmj8n8z0h > > FLIP-426 > > < > https://lists.apache.org/thread/wxn1j848fnfkqsnrs947wh1wmj8n8z0hFLIP-426>: > > https://lists.apache.org/thread/bt931focfl9971cwq194trmf3pkdsxrf > > FLIP-427 > > < > https://lists.apache.org/thread/bt931focfl9971cwq194trmf3pkdsxrfFLIP-427>: > > https://lists.apache.org/thread/vktfzqvb7t4rltg7fdlsyd9sfdmrc4ft > > FLIP-428 > > < > https://lists.apache.org/thread/vktfzqvb7t4rltg7fdlsyd9sfdmrc4ftFLIP-428>: > > https://lists.apache.org/thread/vr8f91p715ct4lop6b3nr0fh4z5p312b > > > > If you want to talk about the overall architecture, roadmap, milestones > or > > something related with multiple FLIPs, please post it here. Otherwise you > > can discuss some details in separate mails. Let's try to avoid repeated > > discussion in different threads. I will sync important messages here if > > there are any in the above threads. > > > > And reply to @Jeyhun: We will ensure the content between those FLIPs is > > consistent. > > > > > > Best, > > Zakelly > > > > On Thu, Mar 7, 2024 at 2:16 PM Yuan Mei wrote: > > > > > I have been a bit busy these few weeks and sorry for responding late. > > > > > > The original thinking of keeping discussion within one thread is for > > easier > > > tracking and avoid for repeated discussion in different threads. > > > > > > For details, It might be good to start in different threads if needed. > > > > > > We will think of a way to better organize the discussion. > > > > > > Best > > > Yuan > > > > > > > > > On Thu, Mar 7, 2024 at 4:38 AM Jeyhun Karimov > > > wrote: > > > > > > > Hi, > > > > > >
[jira] [Created] (FLINK-34724) Fix broken web link in document of legacy Flink CDC Sources
Hongshun Wang created FLINK-34724: - Summary: Fix broken web link in document of legacy Flink CDC Sources Key: FLINK-34724 URL: https://issues.apache.org/jira/browse/FLINK-34724 Project: Flink Issue Type: Improvement Components: Documentation, Flink CDC Reporter: Hongshun Wang Fix For: cdc-3.1.0 In current document, many link is broken due to directory structure changes, such as: * http://localhost:1313/flink/flink-cdc-docs-master/docs/connectors/legacy-flink-cdc-sources/overview/mongodb-cdc.md * https://repo1.maven.org/maven2/org/apache/flink/flink-sql-connector-mongodb-cdc/3.0-SNAPSHOT/flink-sql-connector-mongodb-cdc-3.0-SNAPSHOT.jar -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congrats! Thanks to everyone involved! Best, Zhongqiang Gong Lincoln Lee 于2024年3月18日周一 16:27写道： > The Apache Flink community is very happy to announce the release of Apache > Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 series. > > Apache Flink® is an open-source stream processing framework for > distributed, high-performing, always-available, and accurate data streaming > applications. > > The release is available for download at: > https://flink.apache.org/downloads.html > > Please check out the release blog post for an overview of the improvements > for this bugfix release: > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > The full release notes are available in Jira: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > > We would like to thank all contributors of the Apache Flink community who > made this release possible! > > > Best, > Yun, Jing, Martijn and Lincoln >
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations Best, Weihua On Tue, Mar 19, 2024 at 10:56 AM Rodrigo Meneses wrote: > Congratulations > > On Mon, Mar 18, 2024 at 7:43 PM Yu Chen wrote: > > > Congratulations! > > Thanks to release managers and everyone involved! > > > > Best, > > Yu Chen > > > > > > > 2024年3月19日 01:01，Jeyhun Karimov 写道： > > > > > > Congrats! > > > Thanks to release managers and everyone involved. > > > > > > Regards, > > > Jeyhun > > > > > > On Mon, Mar 18, 2024 at 9:25 AM Lincoln Lee > > wrote: > > > > > >> The Apache Flink community is very happy to announce the release of > > Apache > > >> Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 > > series. > > >> > > >> Apache Flink® is an open-source stream processing framework for > > >> distributed, high-performing, always-available, and accurate data > > streaming > > >> applications. > > >> > > >> The release is available for download at: > > >> https://flink.apache.org/downloads.html > > >> > > >> Please check out the release blog post for an overview of the > > improvements > > >> for this bugfix release: > > >> > > >> > > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > >> > > >> The full release notes are available in Jira: > > >> > > >> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > > >> > > >> We would like to thank all contributors of the Apache Flink community > > who > > >> made this release possible! > > >> > > >> > > >> Best, > > >> Yun, Jing, Martijn and Lincoln > > >> > > > > >
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations On Mon, Mar 18, 2024 at 7:43 PM Yu Chen wrote: > Congratulations! > Thanks to release managers and everyone involved! > > Best, > Yu Chen > > > > 2024年3月19日 01:01，Jeyhun Karimov 写道： > > > > Congrats! > > Thanks to release managers and everyone involved. > > > > Regards, > > Jeyhun > > > > On Mon, Mar 18, 2024 at 9:25 AM Lincoln Lee > wrote: > > > >> The Apache Flink community is very happy to announce the release of > Apache > >> Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 > series. > >> > >> Apache Flink® is an open-source stream processing framework for > >> distributed, high-performing, always-available, and accurate data > streaming > >> applications. > >> > >> The release is available for download at: > >> https://flink.apache.org/downloads.html > >> > >> Please check out the release blog post for an overview of the > improvements > >> for this bugfix release: > >> > >> > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > >> > >> The full release notes are available in Jira: > >> > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > >> > >> We would like to thank all contributors of the Apache Flink community > who > >> made this release possible! > >> > >> > >> Best, > >> Yun, Jing, Martijn and Lincoln > >> > >
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations! Thanks to release managers and everyone involved! Best, Yu Chen > 2024年3月19日 01:01，Jeyhun Karimov 写道： > > Congrats! > Thanks to release managers and everyone involved. > > Regards, > Jeyhun > > On Mon, Mar 18, 2024 at 9:25 AM Lincoln Lee wrote: > >> The Apache Flink community is very happy to announce the release of Apache >> Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 series. >> >> Apache Flink® is an open-source stream processing framework for >> distributed, high-performing, always-available, and accurate data streaming >> applications. >> >> The release is available for download at: >> https://flink.apache.org/downloads.html >> >> Please check out the release blog post for an overview of the improvements >> for this bugfix release: >> >> https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ >> >> The full release notes are available in Jira: >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 >> >> We would like to thank all contributors of the Apache Flink community who >> made this release possible! >> >> >> Best, >> Yun, Jing, Martijn and Lincoln >>
Re: [DISCUSS] FLIP-437: Support ML Models in Flink SQL
Hi Hao,
> I meant how the table name
in window TVF gets translated to `SqlCallingBinding`. Probably we need to
fetch the table definition from the catalog somewhere. Do we treat those
window TVF specially in parser/planner so that catalog is looked up when
they are seen?
The table names are resolved and validated by Calcite SqlValidator. We
don' need to fetch from catalog manually.
The specific checking logic of cumulate window happens in
SqlCumulateTableFunction.OperandMetadataImpl#checkOperandTypes.
The return type of SqlCumulateTableFunction is defined in
#getRowTypeInference() method.
Both are public interfaces provided by Calcite and it seems it's not
specially handled in parser/planner.
I didn't try that, but my gut feeling is that the framework is ready to
extend a customized TVF.
> For what model is, I'm wondering if it has to be datatype or relation. Can
it be another kind of citizen parallel to datatype/relation/function/db?
Redshift also supports `show models` operation, so it seems it's treated
specially as well?
If it is an entity only used in catalog scope (e.g., show xxx, create xxx,
drop xxx), it is fine to introduce it.
We have introduced such one before, called Module: "load module", "show
modules" [1].
But if we want to use Model in TVF parameters, it means it has to be a
relation or datatype, because
that is what it only accepts now.
Thanks for sharing the reason of preferring TVF instead of Redshift way. It
sounds reasonable to me.
Best,
Jark
[1]:
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/modules/
On Fri, 15 Mar 2024 at 13:41, Hao Li wrote:
> Hi Jark,
>
> Thanks for the pointer. Sorry for the confusion: I meant how the table name
> in window TVF gets translated to `SqlCallingBinding`. Probably we need to
> fetch the table definition from the catalog somewhere. Do we treat those
> window TVF specially in parser/planner so that catalog is looked up when
> they are seen?
>
> For what model is, I'm wondering if it has to be datatype or relation. Can
> it be another kind of citizen parallel to datatype/relation/function/db?
> Redshift also supports `show models` operation, so it seems it's treated
> specially as well? The reasons I don't like Redshift's syntax are:
> 1. It's a bit verbose, you need to think of a model name as well as a
> function name and the function name also needs to be unique.
> 2. More importantly, prediction function isn't the only function that can
> operate on models. There could be a set of inference functions [1] and
> evaluation functions [2] which can operate on models. It's hard to specify
> all of them in model creation.
>
> [1]:
>
> https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-predict
> [2]:
>
> https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-evaluate
>
> Thanks,
> Hao
>
> On Thu, Mar 14, 2024 at 8:18 PM Jark Wu wrote:
>
> > Hi Hao,
> >
> > > Can you send me some pointers
> > where the function gets the table information?
> >
> > Here is the code of cumulate window type checking [1].
> >
> > > Also is it possible to support in
> > window functions in addiction to table?
> >
> > Yes. It is not allowed in TVF.
> >
> > Thanks for the syntax links of other systems. The reason I prefer the
> > Redshift way is
> > that it avoids introducing Model as a relation or datatype (referenced
> as a
> > parameter in TVF).
> > Model is not a relation because it can be queried directly (e.g., SELECT
> *
> > FROM model).
> > I'm also confused about making Model as a datatype, because I don't know
> > what class the
> > model parameter of the eval method of TableFunction/ScalarFunction should
> > be. By defining
> > the function with the model, users can directly invoke the function
> without
> > reference to the model name.
> >
> > Best,
> > Jark
> >
> > [1]:
> >
> >
> https://github.com/apache/flink/blob/d6c7eee8243b4fe3e593698f250643534dc79cb5/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/sql/SqlCumulateTableFunction.java#L53
> >
> > On Fri, 15 Mar 2024 at 02:48, Hao Li wrote:
> >
> > > Hi Jark,
> > >
> > > Thanks for the pointers. It's very helpful.
> > >
> > > 1. Looks like `tumble`, `hopping` are keywords in calcite parser. And
> the
> > > syntax `cumulate(Table my_table, ...)` needs to get table information
> > from
> > > catalog somewhere for type validation etc. Can you send me some
> pointers
> > > where the function gets the table information?
> > > 2. The ideal syntax for model function I think would be
> `ML_PREDICT(MODEL
> > > , {table | (query_stmt) })`. I think with
> > special
> > > handling of the `ML_PREDICT` function in parser/planner, maybe we can
> do
> > > this like window functions. But to support `MODEL` keyword, we need
> > calcite
> > > parser change I guess. Also is it possible to support in
> > > window functions in addiction to table?
> > >
> > > For the redshift syntax, I'm not sure the purpose of
[jira] [Created] (FLINK-34723) Parquet writer should restrict map keys to be not null
Xingcan Cui created FLINK-34723:
---
Summary: Parquet writer should restrict map keys to be not null
Key: FLINK-34723
URL: https://issues.apache.org/jira/browse/FLINK-34723
Project: Flink
Issue Type: New Feature
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.18.1, 1.19.0
Reporter: Xingcan Cui
We got the following exception when reading a parquet file (with map types)
generated by Flink.
{code:java}
Map keys must be annotated as required.{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [VOTE] Apache Flink Kubernetes Operator Release 1.8.0, release candidate #1
@Rui @Gyula Thanks for checking the release! >A minor correction is that [3] in the email should point to: >ghcr.io/apache/flink-kubernetes-operator:91d67d9 . But the helm chart and > everything is correct. It's a typo in the vote email. Good catch. Indeed, for the linked Docker image 8938658 points to HEAD^ of the rc branch, 91d67d9 is the HEAD. There are no code changes between those two commits, except for updating the version. So the votes are not impacted, especially because votes are casted against the source release which, as you pointed out, contains the correct image ref. On Mon, Mar 18, 2024 at 9:54 AM Gyula Fóra wrote: > > Hi Max! > > +1 (binding) > > - Verified source release, helm chart + checkpoints / signatures > - Helm points to correct image > - Deployed operator, stateful example and executed upgrade + savepoint > redeploy > - Verified logs > - Flink web PR looks good +1 > > A minor correction is that [3] in the email should point to: > ghcr.io/apache/flink-kubernetes-operator:91d67d9 . But the helm chart and > everything is correct. It's a typo in the vote email. > > Thank you for preparing the release! > > Cheers, > Gyula > > On Mon, Mar 18, 2024 at 8:26 AM Rui Fan <1996fan...@gmail.com> wrote: > > > Thanks Max for driving this release! > > > > +1(non-binding) > > > > - Downloaded artifacts from dist ( svn co > > > > https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.8.0-rc1/ > > ) > > - Verified SHA512 checksums : ( for i in *.tgz; do echo $i; sha512sum > > --check $i.sha512; done ) > > - Verified GPG signatures : ( $ for i in *.tgz; do echo $i; gpg --verify > > $i.asc $i ) > > - Build the source with java-11 and java-17 ( mvn -T 20 clean install > > -DskipTests ) > > - Verified the license header during build the source > > - Verified that chart and appVersion matches the target release (less the > > index.yaml and Chart.yaml ) > > - RC repo works as Helm repo( helm repo add flink-operator-repo-1.8.0-rc1 > > > > https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.8.0-rc1/ > > ) > > - Verified Helm chart can be installed ( helm install > > flink-kubernetes-operator > > flink-operator-repo-1.8.0-rc1/flink-kubernetes-operator --set > > webhook.create=false ) > > - Submitted the autoscaling demo, the autoscaler works well with *memory > > tuning *(kubectl apply -f autoscaling.yaml) > >- job.autoscaler.memory.tuning.enabled: "true" > > - Download Autoscaler standalone: wget > > > > https://repository.apache.org/content/repositories/orgapacheflink-1710/org/apache/flink/flink-autoscaler-standalone/1.8.0/flink-autoscaler-standalone-1.8.0.jar > > - Ran Autoscaler standalone locally, it works well with rescale api and > > JDBC state store/event handler > > > > Best, > > Rui > > > > On Fri, Mar 15, 2024 at 1:45 AM Maximilian Michels wrote: > > > > > Hi everyone, > > > > > > Please review and vote on the release candidate #1 for the version > > > 1.8.0 of the Apache Flink Kubernetes Operator, as follows: > > > > > > [ ] +1, Approve the release > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > **Release Overview** > > > > > > As an overview, the release consists of the following: > > > a) Kubernetes Operator canonical source distribution (including the > > > Dockerfile), to be deployed to the release repository at > > > dist.apache.org > > > b) Kubernetes Operator Helm Chart to be deployed to the release > > > repository at dist.apache.org > > > c) Maven artifacts to be deployed to the Maven Central Repository > > > d) Docker image to be pushed to Dockerhub > > > > > > **Staging Areas to Review** > > > > > > The staging areas containing the above mentioned artifacts are as > > > follows, for your review: > > > * All artifacts for (a), (b) can be found in the corresponding dev > > > repository at dist.apache.org [1] > > > * All artifacts for (c) can be found at the Apache Nexus Repository [2] > > > * The docker image for (d) is staged on github [3] > > > > > > All artifacts are signed with the key > > > DA359CBFCEB13FC302A8793FB655E6F7693D5FDE [4] > > > > > > Other links for your review: > > > * JIRA release notes [5] > > > * source code tag "release-1.8.0-rc1" [6] > > > * PR to update the website Downloads page to include Kubernetes > > > Operator links [7] > > > > > > **Vote Duration** > > > > > > The voting time will run for at least 72 hours. It is adopted by > > > majority approval, with at least 3 PMC affirmative votes. > > > > > > **Note on Verification** > > > > > > You can follow the basic verification guide here [8]. Note that you > > > don't need to verify everything yourself, but please make note of what > > > you have tested together with your +- vote. > > > > > > Thanks, > > > Max > > > > > > [1] > > > > > https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.8.0-rc1/ > > > [2] > > > https://repository.apache.org/content/repositories/orgapacheflink-1710/ > > >
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congrats! Thanks to release managers and everyone involved. Regards, Jeyhun On Mon, Mar 18, 2024 at 9:25 AM Lincoln Lee wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 series. > > Apache Flink® is an open-source stream processing framework for > distributed, high-performing, always-available, and accurate data streaming > applications. > > The release is available for download at: > https://flink.apache.org/downloads.html > > Please check out the release blog post for an overview of the improvements > for this bugfix release: > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > The full release notes are available in Jira: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > > We would like to thank all contributors of the Apache Flink community who > made this release possible! > > > Best, > Yun, Jing, Martijn and Lincoln >
[jira] [Created] (FLINK-34722) Support conditional upserts with Postgres JDBC sink
Pietro created FLINK-34722:
--
Summary: Support conditional upserts with Postgres JDBC sink
Key: FLINK-34722
URL: https://issues.apache.org/jira/browse/FLINK-34722
Project: Flink
Issue Type: Improvement
Components: Connectors / JDBC
Affects Versions: jdbc-3.1.2
Reporter: Pietro
The default Postgres dialect used by the JDBC sink for PostgreSQL DBs does not
support custom _WHERE_ conditions inside upsert statements at the moment.
Indeed, upsert statements returned by the
{{[getUpsertStatement()|https://github.com/apache/flink-connector-jdbc/blob/95294ffbc57c93c2af34cda94c27fc5781e06177/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/dialect/AbstractPostgresCompatibleDialect.java#L61]}}
method are limited to:
{code:sql}
ON CONFLICT (col1, ..., colN)
DO UPDATE SET (col1=EXCLUDED.col1, ..., colN=EXCLUDED.colN)
{code}
PostgreSQL allows a finer-grained control of upsert statements by specifying a
_WHERE_ statement (see [ON CONFLICT
Clause|https://www.postgresql.org/docs/current/sql-insert.html#SQL-ON-CONFLICT]),
for instance:
{code:sql}
ON CONFLICT (col1, ..., colN)
DO UPDATE SET (col1=EXCLUDED.col1, ..., colN=EXCLUDED.colN)
WHERE colN < EXCLUDED.colN
{code}
This could be useful in many use cases, for instance, in a CDC scenario where a
batch reconciliation process has written records in the destination, which now
face the risk of being overwritten by late arriving, stale, records in the
streaming pipeline (adding a condition on the operation timestamp could protect
from these events).
My proposal is to extend the
{{[AbstractPostgresCompatibleDialect|https://github.com/apache/flink-connector-jdbc/blob/main/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/dialect/AbstractPostgresCompatibleDialect.java]}}
functionalities by making the upsert query support _WHERE_ statements provided
by users.
I'm thinking of two possible approaches, but I'd love to hear your opinion on
this:
# provide the statement through options of the JDBC sink connector.
# allow users to plug custom dialects without them having to rewrite the whole
JDBC sink (about this I'll open a separate issue soon)
Thanks for your consideration
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations! Best Regards Peter Huang On Mon, Mar 18, 2024 at 9:28 AM David Radley wrote: > Congratulations! > Kind regards, David > > From: Ahmed Hamdy > Date: Monday, 18 March 2024 at 15:55 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: [ANNOUNCE] Apache Flink 1.19.0 released > Congratulations! > Best Regards > Ahmed Hamdy > > > On Mon, 18 Mar 2024 at 12:30, Xintong Song wrote: > > > Congratulations~! > > > > Best, > > > > Xintong > > > > > > > > On Mon, Mar 18, 2024 at 7:02 PM Feng Jin wrote: > > > > > Congratulations! > > > > > > Best, > > > Feng > > > > > > On Mon, Mar 18, 2024 at 6:18 PM Yuepeng Pan > > wrote: > > > > > > > Congratulations! > > > > > > > > > > > > Thanks to release managers and everyone involved. > > > > > > > > > > > > > > > > > > > > Best, > > > > Yuepeng Pan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 2024-03-18 18:09:45, "Yubin Li" wrote: > > > > >Congratulations! > > > > > > > > > >Thanks to release managers and everyone involved. > > > > > > > > > >On Mon, Mar 18, 2024 at 5:55 PM Hangxiang Yu > > > wrote: > > > > >> > > > > >> Congratulations! > > > > >> Thanks release managers and all involved! > > > > >> > > > > >> On Mon, Mar 18, 2024 at 5:23 PM Hang Ruan > > > > > wrote: > > > > >> > > > > >> > Congratulations! > > > > >> > > > > > >> > Best, > > > > >> > Hang > > > > >> > > > > > >> > Paul Lam 于2024年3月18日周一 17:18写道： > > > > >> > > > > > >> > > Congrats! Thanks to everyone involved! > > > > >> > > > > > > >> > > Best, > > > > >> > > Paul Lam > > > > >> > > > > > > >> > > > 2024年3月18日 16:37，Samrat Deb 写道： > > > > >> > > > > > > > >> > > > Congratulations ! > > > > >> > > > > > > > >> > > > On Mon, 18 Mar 2024 at 2:07 PM, Jingsong Li < > > > > jingsongl...@gmail.com> > > > > >> > > wrote: > > > > >> > > > > > > > >> > > >> Congratulations! > > > > >> > > >> > > > > >> > > >> On Mon, Mar 18, 2024 at 4:30 PM Rui Fan < > > 1996fan...@gmail.com> > > > > wrote: > > > > >> > > >>> > > > > >> > > >>> Congratulations, thanks for the great work! > > > > >> > > >>> > > > > >> > > >>> Best, > > > > >> > > >>> Rui > > > > >> > > >>> > > > > >> > > >>> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee < > > > > lincoln.8...@gmail.com> > > > > >> > > >> wrote: > > > > >> > > > > > > >> > > The Apache Flink community is very happy to announce the > > > > release of > > > > >> > > >> Apache Flink 1.19.0, which is the fisrt release for the > > Apache > > > > Flink > > > > >> > > 1.19 > > > > >> > > >> series. > > > > >> > > > > > > >> > > Apache Flink® is an open-source stream processing > framework > > > for > > > > >> > > >> distributed, high-performing, always-available, and > accurate > > > data > > > > >> > > streaming > > > > >> > > >> applications. > > > > >> > > > > > > >> > > The release is available for download at: > > > > >> > > https://flink.apache.org/downloads.html > > > > >> > > > > > > >> > > Please check out the release blog post for an overview of > > the > > > > >> > > >> improvements for this bugfix release: > > > > >> > > > > > > >> > > >> > > > > >> > > > > > > >> > > > > > > > > > > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > > > >> > > > > > > >> > > The full release notes are available in Jira: > > > > >> > > > > > > >> > > >> > > > > >> > > > > > > >> > > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > > > > >> > > > > > > >> > > We would like to thank all contributors of the Apache > Flink > > > > >> > community > > > > >> > > >> who made this release possible! > > > > >> > > > > > > >> > > > > > > >> > > Best, > > > > >> > > Yun, Jing, Martijn and Lincoln > > > > >> > > >> > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > >> -- > > > > >> Best, > > > > >> Hangxiang. > > > > > > > > > > > Unless otherwise stated above: > > IBM United Kingdom Limited > Registered in England and Wales with number 741598 > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU >
RE: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations! Kind regards, David From: Ahmed Hamdy Date: Monday, 18 March 2024 at 15:55 To: dev@flink.apache.org Subject: [EXTERNAL] Re: [ANNOUNCE] Apache Flink 1.19.0 released Congratulations! Best Regards Ahmed Hamdy On Mon, 18 Mar 2024 at 12:30, Xintong Song wrote: > Congratulations~! > > Best, > > Xintong > > > > On Mon, Mar 18, 2024 at 7:02 PM Feng Jin wrote: > > > Congratulations! > > > > Best, > > Feng > > > > On Mon, Mar 18, 2024 at 6:18 PM Yuepeng Pan > wrote: > > > > > Congratulations! > > > > > > > > > Thanks to release managers and everyone involved. > > > > > > > > > > > > > > > Best, > > > Yuepeng Pan > > > > > > > > > > > > > > > > > > > > > > > > > > > At 2024-03-18 18:09:45, "Yubin Li" wrote: > > > >Congratulations! > > > > > > > >Thanks to release managers and everyone involved. > > > > > > > >On Mon, Mar 18, 2024 at 5:55 PM Hangxiang Yu > > wrote: > > > >> > > > >> Congratulations! > > > >> Thanks release managers and all involved! > > > >> > > > >> On Mon, Mar 18, 2024 at 5:23 PM Hang Ruan > > > wrote: > > > >> > > > >> > Congratulations! > > > >> > > > > >> > Best, > > > >> > Hang > > > >> > > > > >> > Paul Lam 于2024年3月18日周一 17:18写道： > > > >> > > > > >> > > Congrats! Thanks to everyone involved! > > > >> > > > > > >> > > Best, > > > >> > > Paul Lam > > > >> > > > > > >> > > > 2024年3月18日 16:37，Samrat Deb 写道： > > > >> > > > > > > >> > > > Congratulations ! > > > >> > > > > > > >> > > > On Mon, 18 Mar 2024 at 2:07 PM, Jingsong Li < > > > jingsongl...@gmail.com> > > > >> > > wrote: > > > >> > > > > > > >> > > >> Congratulations! > > > >> > > >> > > > >> > > >> On Mon, Mar 18, 2024 at 4:30 PM Rui Fan < > 1996fan...@gmail.com> > > > wrote: > > > >> > > >>> > > > >> > > >>> Congratulations, thanks for the great work! > > > >> > > >>> > > > >> > > >>> Best, > > > >> > > >>> Rui > > > >> > > >>> > > > >> > > >>> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee < > > > lincoln.8...@gmail.com> > > > >> > > >> wrote: > > > >> > > > > > >> > > The Apache Flink community is very happy to announce the > > > release of > > > >> > > >> Apache Flink 1.19.0, which is the fisrt release for the > Apache > > > Flink > > > >> > > 1.19 > > > >> > > >> series. > > > >> > > > > > >> > > Apache Flink® is an open-source stream processing framework > > for > > > >> > > >> distributed, high-performing, always-available, and accurate > > data > > > >> > > streaming > > > >> > > >> applications. > > > >> > > > > > >> > > The release is available for download at: > > > >> > > https://flink.apache.org/downloads.html > > > >> > > > > > >> > > Please check out the release blog post for an overview of > the > > > >> > > >> improvements for this bugfix release: > > > >> > > > > > >> > > >> > > > >> > > > > > >> > > > > > > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > > >> > > > > > >> > > The full release notes are available in Jira: > > > >> > > > > > >> > > >> > > > >> > > > > > >> > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > > > >> > > > > > >> > > We would like to thank all contributors of the Apache Flink > > > >> > community > > > >> > > >> who made this release possible! > > > >> > > > > > >> > > > > > >> > > Best, > > > >> > > Yun, Jing, Martijn and Lincoln > > > >> > > >> > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > >> -- > > > >> Best, > > > >> Hangxiang. > > > > > > Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations! Best Regards Ahmed Hamdy On Mon, 18 Mar 2024 at 12:30, Xintong Song wrote: > Congratulations~! > > Best, > > Xintong > > > > On Mon, Mar 18, 2024 at 7:02 PM Feng Jin wrote: > > > Congratulations! > > > > Best, > > Feng > > > > On Mon, Mar 18, 2024 at 6:18 PM Yuepeng Pan > wrote: > > > > > Congratulations! > > > > > > > > > Thanks to release managers and everyone involved. > > > > > > > > > > > > > > > Best, > > > Yuepeng Pan > > > > > > > > > > > > > > > > > > > > > > > > > > > At 2024-03-18 18:09:45, "Yubin Li" wrote: > > > >Congratulations! > > > > > > > >Thanks to release managers and everyone involved. > > > > > > > >On Mon, Mar 18, 2024 at 5:55 PM Hangxiang Yu > > wrote: > > > >> > > > >> Congratulations! > > > >> Thanks release managers and all involved! > > > >> > > > >> On Mon, Mar 18, 2024 at 5:23 PM Hang Ruan > > > wrote: > > > >> > > > >> > Congratulations! > > > >> > > > > >> > Best, > > > >> > Hang > > > >> > > > > >> > Paul Lam 于2024年3月18日周一 17:18写道： > > > >> > > > > >> > > Congrats! Thanks to everyone involved! > > > >> > > > > > >> > > Best, > > > >> > > Paul Lam > > > >> > > > > > >> > > > 2024年3月18日 16:37，Samrat Deb 写道： > > > >> > > > > > > >> > > > Congratulations ! > > > >> > > > > > > >> > > > On Mon, 18 Mar 2024 at 2:07 PM, Jingsong Li < > > > jingsongl...@gmail.com> > > > >> > > wrote: > > > >> > > > > > > >> > > >> Congratulations! > > > >> > > >> > > > >> > > >> On Mon, Mar 18, 2024 at 4:30 PM Rui Fan < > 1996fan...@gmail.com> > > > wrote: > > > >> > > >>> > > > >> > > >>> Congratulations, thanks for the great work! > > > >> > > >>> > > > >> > > >>> Best, > > > >> > > >>> Rui > > > >> > > >>> > > > >> > > >>> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee < > > > lincoln.8...@gmail.com> > > > >> > > >> wrote: > > > >> > > > > > >> > > The Apache Flink community is very happy to announce the > > > release of > > > >> > > >> Apache Flink 1.19.0, which is the fisrt release for the > Apache > > > Flink > > > >> > > 1.19 > > > >> > > >> series. > > > >> > > > > > >> > > Apache Flink® is an open-source stream processing framework > > for > > > >> > > >> distributed, high-performing, always-available, and accurate > > data > > > >> > > streaming > > > >> > > >> applications. > > > >> > > > > > >> > > The release is available for download at: > > > >> > > https://flink.apache.org/downloads.html > > > >> > > > > > >> > > Please check out the release blog post for an overview of > the > > > >> > > >> improvements for this bugfix release: > > > >> > > > > > >> > > >> > > > >> > > > > > >> > > > > > > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > > >> > > > > > >> > > The full release notes are available in Jira: > > > >> > > > > > >> > > >> > > > >> > > > > > >> > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > > > >> > > > > > >> > > We would like to thank all contributors of the Apache Flink > > > >> > community > > > >> > > >> who made this release possible! > > > >> > > > > > >> > > > > > >> > > Best, > > > >> > > Yun, Jing, Martijn and Lincoln > > > >> > > >> > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > >> -- > > > >> Best, > > > >> Hangxiang. > > > > > >
[jira] [Created] (FLINK-34721) An grammar error occurs in the Barrier test file
Haifei Chen created FLINK-34721: --- Summary: An grammar error occurs in the Barrier test file Key: FLINK-34721 URL: https://issues.apache.org/jira/browse/FLINK-34721 Project: Flink Issue Type: Improvement Components: Tests Reporter: Haifei Chen Attachments: incorrect grammar.png In the flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/io/checkpointing/BarrierAlignmentUtilTest.java file, there is a grammar error in the description of a thrown Exception when testing BarrierAlignmentUtil.DelayableTimer -- This message was sent by Atlassian Jira (v8.20.10#820010)
Wiki permissions
Hello! I updated some instructions that are useful for the apache-flink.slack.com #builds channel -- you used to be able to get instructions how to setup the Redirector extension from an old Slack comment. Since we only have a limited history with the free plan, I copied some into the Canvas for the channel. Would it be possible to give me edit permissions on the Flink wiki[1]? My committer ID is rskraba (tho' I am not a Flink committer). Thanks, Ryan [1]: https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
[jira] [Created] (FLINK-34720) Deploy Maven Snapshot failed on AZP
Ryan Skraba created FLINK-34720:
---
Summary: Deploy Maven Snapshot failed on AZP
Key: FLINK-34720
URL: https://issues.apache.org/jira/browse/FLINK-34720
Project: Flink
Issue Type: Bug
Components: Build System / CI
Affects Versions: 1.20.0
Reporter: Ryan Skraba
There isn't any obvious reason that {{mvn: command not found}} could have
occurred, but we saw it twice this weekend.
*
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58352=logs=eca6b3a6-1600-56cc-916a-c549b3cde3ff=7b3c1df5-9194-5183-5ebd-5567f52d5f8f]
*
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58359=logs=eca6b3a6-1600-56cc-916a-c549b3cde3ff=7b3c1df5-9194-5183-5ebd-5567f52d5f8f=36]
{code:java}
+ [[ tools != \t\o\o\l\s ]]
+ cd ..
+ echo 'Deploying to repository.apache.org'
+ COMMON_OPTIONS='-Prelease,docs-and-source -DskipTests
-DretryFailedDeploymentCount=10 -Dmaven.repo.local=/__w/1/.m2/repository
-Dmaven.wagon.http.pool=false -Dorg.slf4j.simpleLogger.showDateTime=true
-Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss.SSS
-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn
--no-snapshot-updates -B -Dgpg.skip -Drat.skip -Dcheckstyle.skip --settings
/__w/1/s/tools/deploy-settings.xml'
+ mvn clean deploy -Prelease,docs-and-source -DskipTests
-DretryFailedDeploymentCount=10 -Dmaven.repo.local=/__w/1/.m2/repository
-Dmaven.wagon.http.pool=false -Dorg.slf4j.simpleLogger.showDateTime=true
-Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss.SSS
-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn
--no-snapshot-updates -B -Dgpg.skip -Drat.skip -Dcheckstyle.skip --settings
/__w/1/s/tools/deploy-settings.xml
Deploying to repository.apache.org
./releasing/deploy_staging_jars.sh: line 46: mvn: command not found
##[error]Bash exited with code '127'.
Finishing: Deploy maven snapshot
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34719) StreamRecordTest#testWithTimestamp fails on Azure
Ryan Skraba created FLINK-34719:
---
Summary: StreamRecordTest#testWithTimestamp fails on Azure
Key: FLINK-34719
URL: https://issues.apache.org/jira/browse/FLINK-34719
Project: Flink
Issue Type: Bug
Components: Tests
Affects Versions: 1.20.0
Reporter: Ryan Skraba
The ClassCastException *message* expected in StreamRecordTest#testWithTimestamp
as well as StreamRecordTest#testWithNoTimestamp fails on JDK 11, 17, and 21
*
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58352=logs=f0ac5c25-1168-55a5-07ff-0e88223afed9=50bf7a25-bdc4-5e56-5478-c7b4511dde53=10341]
*
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58352=logs=675bf62c-8558-587e-2555-dcad13acefb5=5878eed3-cc1e-5b12-1ed0-9e7139ce0992=9828]
*
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58352=logs=d06b80b4-9e88-5d40-12a2-18072cf60528=609ecd5a-3f6e-5d0c-2239-2096b155a4d0=9833]
{code:java}
Expecting throwable message:
Mar 16 01:35:07 "class
org.apache.flink.streaming.runtime.streamrecord.StreamRecord cannot be cast to
class org.apache.flink.streaming.api.watermark.Watermark
(org.apache.flink.streaming.runtime.streamrecord.StreamRecord and
org.apache.flink.streaming.api.watermark.Watermark are in unnamed module of
loader 'app')"
Mar 16 01:35:07 to contain:
Mar 16 01:35:07 "cannot be cast to
org.apache.flink.streaming.api.watermark.Watermark"
Mar 16 01:35:07 but did not.
Mar 16 01:35:07
Mar 16 01:35:07 Throwable that failed the check:
Mar 16 01:35:07
Mar 16 01:35:07 java.lang.ClassCastException: class
org.apache.flink.streaming.runtime.streamrecord.StreamRecord cannot be cast to
class org.apache.flink.streaming.api.watermark.Watermark
(org.apache.flink.streaming.runtime.streamrecord.StreamRecord and
org.apache.flink.streaming.api.watermark.Watermark are in unnamed module of
loader 'app')
Mar 16 01:35:07 at
org.apache.flink.streaming.runtime.streamrecord.StreamElement.asWatermark(StreamElement.java:92)
Mar 16 01:35:07 at
org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
Mar 16 01:35:07 at
org.assertj.core.api.AssertionsForClassTypes.catchThrowable(AssertionsForClassTypes.java:892)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34718) PartitionedWindowed
Ryan Skraba created FLINK-34718:
---
Summary: PartitionedWindowed
Key: FLINK-34718
URL: https://issues.apache.org/jira/browse/FLINK-34718
Project: Flink
Issue Type: Bug
Components: API / DataStream
Affects Versions: 1.20.0
Reporter: Ryan Skraba
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58320=logs=8fd9202e-fd17-5b26-353c-ac1ff76c8f28=ea7cf968-e585-52cb-e0fc-f48de023a7ca=9646]
18 of the KeyedPartitionWindowedStreamITCase and
NonKeyedPartitionWindowedStreamITCase unit tests introduced in FLINK-34543 are
failing in the adaptive scheduler profile, with errors similar to:
{code:java}
Mar 15 01:54:12 Caused by: java.lang.IllegalStateException: The adaptive
scheduler supports pipelined data exchanges (violated by MapPartition
(org.apache.flink.streaming.runtime.tasks.OneInputStreamTask) ->
ddb598ad156ed281023ba4eebbe487e3).
Mar 15 01:54:12 at
org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
Mar 15 01:54:12 at
org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.assertPreconditions(AdaptiveScheduler.java:438)
Mar 15 01:54:12 at
org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.(AdaptiveScheduler.java:356)
Mar 15 01:54:12 at
org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerFactory.createInstance(AdaptiveSchedulerFactory.java:124)
Mar 15 01:54:12 at
org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:121)
Mar 15 01:54:12 at
org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:384)
Mar 15 01:54:12 at
org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:361)
Mar 15 01:54:12 at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128)
Mar 15 01:54:12 at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100)
Mar 15 01:54:12 at
org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
Mar 15 01:54:12 at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
Mar 15 01:54:12 ... 4 more
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations~! Best, Xintong On Mon, Mar 18, 2024 at 7:02 PM Feng Jin wrote: > Congratulations! > > Best, > Feng > > On Mon, Mar 18, 2024 at 6:18 PM Yuepeng Pan wrote: > > > Congratulations! > > > > > > Thanks to release managers and everyone involved. > > > > > > > > > > Best, > > Yuepeng Pan > > > > > > > > > > > > > > > > > > At 2024-03-18 18:09:45, "Yubin Li" wrote: > > >Congratulations! > > > > > >Thanks to release managers and everyone involved. > > > > > >On Mon, Mar 18, 2024 at 5:55 PM Hangxiang Yu > wrote: > > >> > > >> Congratulations! > > >> Thanks release managers and all involved! > > >> > > >> On Mon, Mar 18, 2024 at 5:23 PM Hang Ruan > > wrote: > > >> > > >> > Congratulations! > > >> > > > >> > Best, > > >> > Hang > > >> > > > >> > Paul Lam 于2024年3月18日周一 17:18写道： > > >> > > > >> > > Congrats! Thanks to everyone involved! > > >> > > > > >> > > Best, > > >> > > Paul Lam > > >> > > > > >> > > > 2024年3月18日 16:37，Samrat Deb 写道： > > >> > > > > > >> > > > Congratulations ! > > >> > > > > > >> > > > On Mon, 18 Mar 2024 at 2:07 PM, Jingsong Li < > > jingsongl...@gmail.com> > > >> > > wrote: > > >> > > > > > >> > > >> Congratulations! > > >> > > >> > > >> > > >> On Mon, Mar 18, 2024 at 4:30 PM Rui Fan <1996fan...@gmail.com> > > wrote: > > >> > > >>> > > >> > > >>> Congratulations, thanks for the great work! > > >> > > >>> > > >> > > >>> Best, > > >> > > >>> Rui > > >> > > >>> > > >> > > >>> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee < > > lincoln.8...@gmail.com> > > >> > > >> wrote: > > >> > > > > >> > > The Apache Flink community is very happy to announce the > > release of > > >> > > >> Apache Flink 1.19.0, which is the fisrt release for the Apache > > Flink > > >> > > 1.19 > > >> > > >> series. > > >> > > > > >> > > Apache Flink® is an open-source stream processing framework > for > > >> > > >> distributed, high-performing, always-available, and accurate > data > > >> > > streaming > > >> > > >> applications. > > >> > > > > >> > > The release is available for download at: > > >> > > https://flink.apache.org/downloads.html > > >> > > > > >> > > Please check out the release blog post for an overview of the > > >> > > >> improvements for this bugfix release: > > >> > > > > >> > > >> > > >> > > > > >> > > > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > >> > > > > >> > > The full release notes are available in Jira: > > >> > > > > >> > > >> > > >> > > > > >> > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > > >> > > > > >> > > We would like to thank all contributors of the Apache Flink > > >> > community > > >> > > >> who made this release possible! > > >> > > > > >> > > > > >> > > Best, > > >> > > Yun, Jing, Martijn and Lincoln > > >> > > >> > > >> > > > > >> > > > > >> > > > >> > > >> > > >> -- > > >> Best, > > >> Hangxiang. > > >
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations Best, Ron Yanfei Lei 于2024年3月18日周一 20:01写道： > Congrats, thanks for the great work! > > Sergey Nuyanzin 于2024年3月18日周一 19:30写道： > > > > Congratulations, thanks release managers and everyone involved for the > great work! > > > > On Mon, Mar 18, 2024 at 12:15 PM Benchao Li > wrote: > >> > >> Congratulations! And thanks to all release managers and everyone > >> involved in this release! > >> > >> Yubin Li 于2024年3月18日周一 18:11写道： > >> > > >> > Congratulations! > >> > > >> > Thanks to release managers and everyone involved. > >> > > >> > On Mon, Mar 18, 2024 at 5:55 PM Hangxiang Yu > wrote: > >> > > > >> > > Congratulations! > >> > > Thanks release managers and all involved! > >> > > > >> > > On Mon, Mar 18, 2024 at 5:23 PM Hang Ruan > wrote: > >> > > > >> > > > Congratulations! > >> > > > > >> > > > Best, > >> > > > Hang > >> > > > > >> > > > Paul Lam 于2024年3月18日周一 17:18写道： > >> > > > > >> > > > > Congrats! Thanks to everyone involved! > >> > > > > > >> > > > > Best, > >> > > > > Paul Lam > >> > > > > > >> > > > > > 2024年3月18日 16:37，Samrat Deb 写道： > >> > > > > > > >> > > > > > Congratulations ! > >> > > > > > > >> > > > > > On Mon, 18 Mar 2024 at 2:07 PM, Jingsong Li < > jingsongl...@gmail.com> > >> > > > > wrote: > >> > > > > > > >> > > > > >> Congratulations! > >> > > > > >> > >> > > > > >> On Mon, Mar 18, 2024 at 4:30 PM Rui Fan < > 1996fan...@gmail.com> wrote: > >> > > > > >>> > >> > > > > >>> Congratulations, thanks for the great work! > >> > > > > >>> > >> > > > > >>> Best, > >> > > > > >>> Rui > >> > > > > >>> > >> > > > > >>> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee < > lincoln.8...@gmail.com> > >> > > > > >> wrote: > >> > > > > > >> > > > > The Apache Flink community is very happy to announce the > release of > >> > > > > >> Apache Flink 1.19.0, which is the fisrt release for the > Apache Flink > >> > > > > 1.19 > >> > > > > >> series. > >> > > > > > >> > > > > Apache Flink® is an open-source stream processing > framework for > >> > > > > >> distributed, high-performing, always-available, and accurate > data > >> > > > > streaming > >> > > > > >> applications. > >> > > > > > >> > > > > The release is available for download at: > >> > > > > https://flink.apache.org/downloads.html > >> > > > > > >> > > > > Please check out the release blog post for an overview of > the > >> > > > > >> improvements for this bugfix release: > >> > > > > > >> > > > > >> > >> > > > > > >> > > > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > >> > > > > > >> > > > > The full release notes are available in Jira: > >> > > > > > >> > > > > >> > >> > > > > > >> > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > >> > > > > > >> > > > > We would like to thank all contributors of the Apache Flink > >> > > > community > >> > > > > >> who made this release possible! > >> > > > > > >> > > > > > >> > > > > Best, > >> > > > > Yun, Jing, Martijn and Lincoln > >> > > > > >> > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > >> > > -- > >> > > Best, > >> > > Hangxiang. > >> > >> > >> > >> -- > >> > >> Best, > >> Benchao Li > > > > > > > > -- > > Best regards, > > Sergey > > > > -- > Best, > Yanfei >
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congrats, thanks for the great work! Sergey Nuyanzin 于2024年3月18日周一 19:30写道： > > Congratulations, thanks release managers and everyone involved for the great > work! > > On Mon, Mar 18, 2024 at 12:15 PM Benchao Li wrote: >> >> Congratulations! And thanks to all release managers and everyone >> involved in this release! >> >> Yubin Li 于2024年3月18日周一 18:11写道： >> > >> > Congratulations! >> > >> > Thanks to release managers and everyone involved. >> > >> > On Mon, Mar 18, 2024 at 5:55 PM Hangxiang Yu wrote: >> > > >> > > Congratulations! >> > > Thanks release managers and all involved! >> > > >> > > On Mon, Mar 18, 2024 at 5:23 PM Hang Ruan wrote: >> > > >> > > > Congratulations! >> > > > >> > > > Best, >> > > > Hang >> > > > >> > > > Paul Lam 于2024年3月18日周一 17:18写道： >> > > > >> > > > > Congrats! Thanks to everyone involved! >> > > > > >> > > > > Best, >> > > > > Paul Lam >> > > > > >> > > > > > 2024年3月18日 16:37，Samrat Deb 写道： >> > > > > > >> > > > > > Congratulations ! >> > > > > > >> > > > > > On Mon, 18 Mar 2024 at 2:07 PM, Jingsong Li >> > > > > > >> > > > > wrote: >> > > > > > >> > > > > >> Congratulations! >> > > > > >> >> > > > > >> On Mon, Mar 18, 2024 at 4:30 PM Rui Fan <1996fan...@gmail.com> >> > > > > >> wrote: >> > > > > >>> >> > > > > >>> Congratulations, thanks for the great work! >> > > > > >>> >> > > > > >>> Best, >> > > > > >>> Rui >> > > > > >>> >> > > > > >>> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee >> > > > > >>> >> > > > > >> wrote: >> > > > > >> > > > > The Apache Flink community is very happy to announce the >> > > > > release of >> > > > > >> Apache Flink 1.19.0, which is the fisrt release for the Apache >> > > > > >> Flink >> > > > > 1.19 >> > > > > >> series. >> > > > > >> > > > > Apache Flink® is an open-source stream processing framework for >> > > > > >> distributed, high-performing, always-available, and accurate data >> > > > > streaming >> > > > > >> applications. >> > > > > >> > > > > The release is available for download at: >> > > > > https://flink.apache.org/downloads.html >> > > > > >> > > > > Please check out the release blog post for an overview of the >> > > > > >> improvements for this bugfix release: >> > > > > >> > > > > >> >> > > > > >> > > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ >> > > > > >> > > > > The full release notes are available in Jira: >> > > > > >> > > > > >> >> > > > > >> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 >> > > > > >> > > > > We would like to thank all contributors of the Apache Flink >> > > > community >> > > > > >> who made this release possible! >> > > > > >> > > > > >> > > > > Best, >> > > > > Yun, Jing, Martijn and Lincoln >> > > > > >> >> > > > > >> > > > > >> > > > >> > > >> > > >> > > -- >> > > Best, >> > > Hangxiang. >> >> >> >> -- >> >> Best, >> Benchao Li > > > > -- > Best regards, > Sergey -- Best, Yanfei
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations, thanks release managers and everyone involved for the great work! On Mon, Mar 18, 2024 at 12:15 PM Benchao Li wrote: > Congratulations! And thanks to all release managers and everyone > involved in this release! > > Yubin Li 于2024年3月18日周一 18:11写道： > > > > Congratulations! > > > > Thanks to release managers and everyone involved. > > > > On Mon, Mar 18, 2024 at 5:55 PM Hangxiang Yu > wrote: > > > > > > Congratulations! > > > Thanks release managers and all involved! > > > > > > On Mon, Mar 18, 2024 at 5:23 PM Hang Ruan > wrote: > > > > > > > Congratulations! > > > > > > > > Best, > > > > Hang > > > > > > > > Paul Lam 于2024年3月18日周一 17:18写道： > > > > > > > > > Congrats! Thanks to everyone involved! > > > > > > > > > > Best, > > > > > Paul Lam > > > > > > > > > > > 2024年3月18日 16:37，Samrat Deb 写道： > > > > > > > > > > > > Congratulations ! > > > > > > > > > > > > On Mon, 18 Mar 2024 at 2:07 PM, Jingsong Li < > jingsongl...@gmail.com> > > > > > wrote: > > > > > > > > > > > >> Congratulations! > > > > > >> > > > > > >> On Mon, Mar 18, 2024 at 4:30 PM Rui Fan <1996fan...@gmail.com> > wrote: > > > > > >>> > > > > > >>> Congratulations, thanks for the great work! > > > > > >>> > > > > > >>> Best, > > > > > >>> Rui > > > > > >>> > > > > > >>> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee < > lincoln.8...@gmail.com> > > > > > >> wrote: > > > > > > > > > > The Apache Flink community is very happy to announce the > release of > > > > > >> Apache Flink 1.19.0, which is the fisrt release for the Apache > Flink > > > > > 1.19 > > > > > >> series. > > > > > > > > > > Apache Flink® is an open-source stream processing framework > for > > > > > >> distributed, high-performing, always-available, and accurate > data > > > > > streaming > > > > > >> applications. > > > > > > > > > > The release is available for download at: > > > > > https://flink.apache.org/downloads.html > > > > > > > > > > Please check out the release blog post for an overview of the > > > > > >> improvements for this bugfix release: > > > > > > > > > > >> > > > > > > > > > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > > > > > > > > > The full release notes are available in Jira: > > > > > > > > > > >> > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > > > > > > > > > > We would like to thank all contributors of the Apache Flink > > > > community > > > > > >> who made this release possible! > > > > > > > > > > > > > > > Best, > > > > > Yun, Jing, Martijn and Lincoln > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > -- > > > Best, > > > Hangxiang. > > > > -- > > Best, > Benchao Li > -- Best regards, Sergey
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations! And thanks to all release managers and everyone involved in this release! Yubin Li 于2024年3月18日周一 18:11写道： > > Congratulations! > > Thanks to release managers and everyone involved. > > On Mon, Mar 18, 2024 at 5:55 PM Hangxiang Yu wrote: > > > > Congratulations! > > Thanks release managers and all involved! > > > > On Mon, Mar 18, 2024 at 5:23 PM Hang Ruan wrote: > > > > > Congratulations! > > > > > > Best, > > > Hang > > > > > > Paul Lam 于2024年3月18日周一 17:18写道： > > > > > > > Congrats! Thanks to everyone involved! > > > > > > > > Best, > > > > Paul Lam > > > > > > > > > 2024年3月18日 16:37，Samrat Deb 写道： > > > > > > > > > > Congratulations ! > > > > > > > > > > On Mon, 18 Mar 2024 at 2:07 PM, Jingsong Li > > > > wrote: > > > > > > > > > >> Congratulations! > > > > >> > > > > >> On Mon, Mar 18, 2024 at 4:30 PM Rui Fan <1996fan...@gmail.com> wrote: > > > > >>> > > > > >>> Congratulations, thanks for the great work! > > > > >>> > > > > >>> Best, > > > > >>> Rui > > > > >>> > > > > >>> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee > > > > >> wrote: > > > > > > > > The Apache Flink community is very happy to announce the release of > > > > >> Apache Flink 1.19.0, which is the fisrt release for the Apache Flink > > > > 1.19 > > > > >> series. > > > > > > > > Apache Flink® is an open-source stream processing framework for > > > > >> distributed, high-performing, always-available, and accurate data > > > > streaming > > > > >> applications. > > > > > > > > The release is available for download at: > > > > https://flink.apache.org/downloads.html > > > > > > > > Please check out the release blog post for an overview of the > > > > >> improvements for this bugfix release: > > > > > > > > >> > > > > > > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > > > > > > > The full release notes are available in Jira: > > > > > > > > >> > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > > > > > > > > We would like to thank all contributors of the Apache Flink > > > community > > > > >> who made this release possible! > > > > > > > > > > > > Best, > > > > Yun, Jing, Martijn and Lincoln > > > > >> > > > > > > > > > > > > > > > > > -- > > Best, > > Hangxiang. -- Best, Benchao Li
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations! Best, Feng On Mon, Mar 18, 2024 at 6:18 PM Yuepeng Pan wrote: > Congratulations! > > > Thanks to release managers and everyone involved. > > > > > Best, > Yuepeng Pan > > > > > > > > > At 2024-03-18 18:09:45, "Yubin Li" wrote: > >Congratulations! > > > >Thanks to release managers and everyone involved. > > > >On Mon, Mar 18, 2024 at 5:55 PM Hangxiang Yu wrote: > >> > >> Congratulations! > >> Thanks release managers and all involved! > >> > >> On Mon, Mar 18, 2024 at 5:23 PM Hang Ruan > wrote: > >> > >> > Congratulations! > >> > > >> > Best, > >> > Hang > >> > > >> > Paul Lam 于2024年3月18日周一 17:18写道： > >> > > >> > > Congrats! Thanks to everyone involved! > >> > > > >> > > Best, > >> > > Paul Lam > >> > > > >> > > > 2024年3月18日 16:37，Samrat Deb 写道： > >> > > > > >> > > > Congratulations ! > >> > > > > >> > > > On Mon, 18 Mar 2024 at 2:07 PM, Jingsong Li < > jingsongl...@gmail.com> > >> > > wrote: > >> > > > > >> > > >> Congratulations! > >> > > >> > >> > > >> On Mon, Mar 18, 2024 at 4:30 PM Rui Fan <1996fan...@gmail.com> > wrote: > >> > > >>> > >> > > >>> Congratulations, thanks for the great work! > >> > > >>> > >> > > >>> Best, > >> > > >>> Rui > >> > > >>> > >> > > >>> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee < > lincoln.8...@gmail.com> > >> > > >> wrote: > >> > > > >> > > The Apache Flink community is very happy to announce the > release of > >> > > >> Apache Flink 1.19.0, which is the fisrt release for the Apache > Flink > >> > > 1.19 > >> > > >> series. > >> > > > >> > > Apache Flink® is an open-source stream processing framework for > >> > > >> distributed, high-performing, always-available, and accurate data > >> > > streaming > >> > > >> applications. > >> > > > >> > > The release is available for download at: > >> > > https://flink.apache.org/downloads.html > >> > > > >> > > Please check out the release blog post for an overview of the > >> > > >> improvements for this bugfix release: > >> > > > >> > > >> > >> > > > >> > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > >> > > > >> > > The full release notes are available in Jira: > >> > > > >> > > >> > >> > > > >> > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > >> > > > >> > > We would like to thank all contributors of the Apache Flink > >> > community > >> > > >> who made this release possible! > >> > > > >> > > > >> > > Best, > >> > > Yun, Jing, Martijn and Lincoln > >> > > >> > >> > > > >> > > > >> > > >> > >> > >> -- > >> Best, > >> Hangxiang. >
[jira] [Created] (FLINK-34717) BroadcastStateITCase failed fatally with 127 exit code
Ryan Skraba created FLINK-34717:
---
Summary: BroadcastStateITCase failed fatally with 127 exit code
Key: FLINK-34717
URL: https://issues.apache.org/jira/browse/FLINK-34717
Project: Flink
Issue Type: Bug
Affects Versions: 1.20.0
Reporter: Ryan Skraba
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58306=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba=9069]
{code:java}
Mar 14 13:58:43 13:58:43.330 [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:3.2.2:test (integration-tests)
on project flink-tests:
Mar 14 13:58:43 13:58:43.330 [ERROR]
Mar 14 13:58:43 13:58:43.330 [ERROR] Please refer to
/__w/1/s/flink-tests/target/surefire-reports for the individual test results.
Mar 14 13:58:43 13:58:43.330 [ERROR] Please refer to dump files (if any exist)
[date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
Mar 14 13:58:43 13:58:43.330 [ERROR] ExecutionException The forked VM
terminated without properly saying goodbye. VM crash or System.exit called?
Mar 14 13:58:43 13:58:43.330 [ERROR] Command was /bin/sh -c cd
'/__w/1/s/flink-tests' && '/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java'
'-XX:+UseG1GC' '-Xms256m' '-XX:+IgnoreUnrecognizedVMOptions'
'--add-opens=java.base/java.util=ALL-UNNAMED'
'--add-opens=java.base/java.io=ALL-UNNAMED' '-Xmx1536m' '-jar'
'/__w/1/s/flink-tests/target/surefire/surefirebooter-20240314132147062_959.jar'
'/__w/1/s/flink-tests/target/surefire' '2024-03-14T13-21-44_122-jvmRun1'
'surefire-20240314132147062_957tmp' 'surefire_254-20240314132147062_958tmp'
Mar 14 13:58:43 13:58:43.330 [ERROR] Error occurred in starting fork, check
output in log
Mar 14 13:58:43 13:58:43.330 [ERROR] Process Exit Code: 127
Mar 14 13:58:43 13:58:43.330 [ERROR]
org.apache.maven.surefire.booter.SurefireBooterForkException:
ExecutionException The forked VM terminated without properly saying goodbye. VM
crash or System.exit called?
Mar 14 13:58:43 13:58:43.330 [ERROR] Command was /bin/sh -c cd
'/__w/1/s/flink-tests' && '/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java'
'-XX:+UseG1GC' '-Xms256m' '-XX:+IgnoreUnrecognizedVMOptions'
'--add-opens=java.base/java.util=ALL-UNNAMED'
'--add-opens=java.base/java.io=ALL-UNNAMED' '-Xmx1536m' '-jar'
'/__w/1/s/flink-tests/target/surefire/surefirebooter-20240314132147062_959.jar'
'/__w/1/s/flink-tests/target/surefire' '2024-03-14T13-21-44_122-jvmRun1'
'surefire-20240314132147062_957tmp' 'surefire_254-20240314132147062_958tmp'
Mar 14 13:58:43 13:58:43.330 [ERROR] Error occurred in starting fork, check
output in log
Mar 14 13:58:43 13:58:43.330 [ERROR] Process Exit Code: 127
Mar 14 13:58:43 13:58:43.330 [ERROR]at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:456)
Mar 14 13:58:43 13:58:43.330 [ERROR]at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:418)
Mar 14 13:58:43 13:58:43.330 [ERROR]at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:297)
Mar 14 13:58:43 13:58:43.331 [ERROR]at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250)
{code}
Looking at the watchdog, only the
org.apache.flink.test.streaming.runtime.BroadcastStateITCase is started without
finished. It has two test methods which are both successfully run, so the
problem might with the {{
{color:#00}MiniClusterWithClientResource{color}}} not shutting down.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34716) Build 1.19 docs in GitHub Action and mark 1.19 as stable in docs
lincoln lee created FLINK-34716: --- Summary: Build 1.19 docs in GitHub Action and mark 1.19 as stable in docs Key: FLINK-34716 URL: https://issues.apache.org/jira/browse/FLINK-34716 Project: Flink Issue Type: Sub-task Reporter: lincoln lee Assignee: lincoln lee -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations! Thanks to release managers and everyone involved. Best, Yuepeng Pan At 2024-03-18 18:09:45, "Yubin Li" wrote: >Congratulations! > >Thanks to release managers and everyone involved. > >On Mon, Mar 18, 2024 at 5:55 PM Hangxiang Yu wrote: >> >> Congratulations! >> Thanks release managers and all involved! >> >> On Mon, Mar 18, 2024 at 5:23 PM Hang Ruan wrote: >> >> > Congratulations! >> > >> > Best, >> > Hang >> > >> > Paul Lam 于2024年3月18日周一 17:18写道： >> > >> > > Congrats! Thanks to everyone involved! >> > > >> > > Best, >> > > Paul Lam >> > > >> > > > 2024年3月18日 16:37，Samrat Deb 写道： >> > > > >> > > > Congratulations ! >> > > > >> > > > On Mon, 18 Mar 2024 at 2:07 PM, Jingsong Li >> > > wrote: >> > > > >> > > >> Congratulations! >> > > >> >> > > >> On Mon, Mar 18, 2024 at 4:30 PM Rui Fan <1996fan...@gmail.com> wrote: >> > > >>> >> > > >>> Congratulations, thanks for the great work! >> > > >>> >> > > >>> Best, >> > > >>> Rui >> > > >>> >> > > >>> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee >> > > >> wrote: >> > > >> > > The Apache Flink community is very happy to announce the release of >> > > >> Apache Flink 1.19.0, which is the fisrt release for the Apache Flink >> > > 1.19 >> > > >> series. >> > > >> > > Apache Flink® is an open-source stream processing framework for >> > > >> distributed, high-performing, always-available, and accurate data >> > > streaming >> > > >> applications. >> > > >> > > The release is available for download at: >> > > https://flink.apache.org/downloads.html >> > > >> > > Please check out the release blog post for an overview of the >> > > >> improvements for this bugfix release: >> > > >> > > >> >> > > >> > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ >> > > >> > > The full release notes are available in Jira: >> > > >> > > >> >> > > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 >> > > >> > > We would like to thank all contributors of the Apache Flink >> > community >> > > >> who made this release possible! >> > > >> > > >> > > Best, >> > > Yun, Jing, Martijn and Lincoln >> > > >> >> > > >> > > >> > >> >> >> -- >> Best, >> Hangxiang.
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations! Thanks to release managers and everyone involved. On Mon, Mar 18, 2024 at 5:55 PM Hangxiang Yu wrote: > > Congratulations! > Thanks release managers and all involved! > > On Mon, Mar 18, 2024 at 5:23 PM Hang Ruan wrote: > > > Congratulations! > > > > Best, > > Hang > > > > Paul Lam 于2024年3月18日周一 17:18写道： > > > > > Congrats! Thanks to everyone involved! > > > > > > Best, > > > Paul Lam > > > > > > > 2024年3月18日 16:37，Samrat Deb 写道： > > > > > > > > Congratulations ! > > > > > > > > On Mon, 18 Mar 2024 at 2:07 PM, Jingsong Li > > > wrote: > > > > > > > >> Congratulations! > > > >> > > > >> On Mon, Mar 18, 2024 at 4:30 PM Rui Fan <1996fan...@gmail.com> wrote: > > > >>> > > > >>> Congratulations, thanks for the great work! > > > >>> > > > >>> Best, > > > >>> Rui > > > >>> > > > >>> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee > > > >> wrote: > > > > > > The Apache Flink community is very happy to announce the release of > > > >> Apache Flink 1.19.0, which is the fisrt release for the Apache Flink > > > 1.19 > > > >> series. > > > > > > Apache Flink® is an open-source stream processing framework for > > > >> distributed, high-performing, always-available, and accurate data > > > streaming > > > >> applications. > > > > > > The release is available for download at: > > > https://flink.apache.org/downloads.html > > > > > > Please check out the release blog post for an overview of the > > > >> improvements for this bugfix release: > > > > > > >> > > > > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > > > > > The full release notes are available in Jira: > > > > > > >> > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > > > > > > We would like to thank all contributors of the Apache Flink > > community > > > >> who made this release possible! > > > > > > > > > Best, > > > Yun, Jing, Martijn and Lincoln > > > >> > > > > > > > > > > > -- > Best, > Hangxiang.
Re: Temporal join on rolling aggregate
Hi everyone, Before digging into what it would it take to implement a general solution, I narrowed down the scope to write a fix which makes the query mentioned in the thread work. Here are some findings: - For the temporal join logic, it's not the watermark that matters but having a TimeIndicatorRelDataType column in the input relation. To address that, in the PR below, we introduced a call to the LAST_VALUE aggregate function to bring a timestamp column to the view. That makes the query works, but we think it is not enough. It would probably require a distinct aggregate function or a new syntax to be able to support more general use cases. - There is a relationship between the way the logical operators are reordered, the way the special Flink's Timestamp time is materialized and the watermark assigner. - I also looked into the flink-sql-parser and I found out that Flink has customized the parsing of the CREATE and DROP statements (https://github.com/apache/flink/blob/master/flink-table/flink-sql-parser/src/main/codegen/data/Parser.tdd#L627-L638) (and Calcite supports as well support customizations for ALTER statements) but Calcite does not seem to support changes to the SELECT syntax (see https://issues.apache.org/jira/browse/CALCITE-4979). I mention it because I think it will inform what could be done syntax-wise. and a PR that highlights the changes with the complete investigation https://github.com/apache/flink/pull/24506 This is more of a demonstration and I am looking to get feedback from someone who has more experience with the codebase. Thanks, Seb On Tue, Mar 5, 2024, at 10:07, Gyula Fóra wrote: > Hi Everyone! > > I have discussed this with Sébastien Chevalley, he is going to prepare and > drive the FLIP while I will assist him along the way. > > Thanks > Gyula > > On Tue, Mar 5, 2024 at 9:57 AM wrote: >> I do agree with Ron Liu. >> This would definitely need a FLIP as it would impact SQL and extend it with >> the equivalent of TimestampAssigners in the Java API. >> >> Is there any existing JIRA here, or is anybody willing to drive a FLIP? >> On Feb 26, 2024 at 02:36 +0100, Ron liu , wrote: >>> +1, >>> But I think this should be a more general requirement, that is, support for >>> declaring watermarks in query, which can be declared for any type of >>> source, such as table, view. Similar to databricks provided [1], this needs >>> a FLIP. >>> >>> [1] >>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-qry-select-watermark.html >>> >>> Best, >>> Ron
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations! Thanks release managers and all involved! On Mon, Mar 18, 2024 at 5:23 PM Hang Ruan wrote: > Congratulations! > > Best, > Hang > > Paul Lam 于2024年3月18日周一 17:18写道： > > > Congrats! Thanks to everyone involved! > > > > Best, > > Paul Lam > > > > > 2024年3月18日 16:37，Samrat Deb 写道： > > > > > > Congratulations ! > > > > > > On Mon, 18 Mar 2024 at 2:07 PM, Jingsong Li > > wrote: > > > > > >> Congratulations! > > >> > > >> On Mon, Mar 18, 2024 at 4:30 PM Rui Fan <1996fan...@gmail.com> wrote: > > >>> > > >>> Congratulations, thanks for the great work! > > >>> > > >>> Best, > > >>> Rui > > >>> > > >>> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee > > >> wrote: > > > > The Apache Flink community is very happy to announce the release of > > >> Apache Flink 1.19.0, which is the fisrt release for the Apache Flink > > 1.19 > > >> series. > > > > Apache Flink® is an open-source stream processing framework for > > >> distributed, high-performing, always-available, and accurate data > > streaming > > >> applications. > > > > The release is available for download at: > > https://flink.apache.org/downloads.html > > > > Please check out the release blog post for an overview of the > > >> improvements for this bugfix release: > > > > >> > > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > > > The full release notes are available in Jira: > > > > >> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > > > > We would like to thank all contributors of the Apache Flink > community > > >> who made this release possible! > > > > > > Best, > > Yun, Jing, Martijn and Lincoln > > >> > > > > > -- Best, Hangxiang.
Re: Re: [DISCUSS] FLIP-436: Introduce "SHOW CREATE CATALOG" Syntax
Hi Jark,
Thanks for your response, I have updated FLIP-436: Introduce
Catalog-related Syntax [1] as you suggested.
If there are no more comments within 24 hours, I will start a vote for
this, thanks :)
Best,
Yubin
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-436%3A+Introduce+Catalog-related+Syntax
On Mon, Mar 18, 2024 at 4:39 PM Jark Wu wrote:
>
> Hi Yubin,
>
> Thanks for the quick response. The suggestion sounds good to me!
>
> Best,
> Jark
>
> On Mon, 18 Mar 2024 at 13:06, Yubin Li wrote:
>
> > Hi Jark,
> >
> > Good pointing! Thanks for your reply, there are some details to align :)
> >
> > 1. I think the purpose of DESCRIBE CATALOG is to display metadata
> > > information including catalog name,
> > > catalog comment (may be introduced in the future), catalog type, and
> > > catalog properties (for example [1])
> >
> > Adopting { DESC | DESCRIBE } CATALOG [ EXTENDED ] xx as formal syntax,
> > Producing rich and compatible results for future needs is very important.
> > When
> > specifying "extended" in the syntax, it will output the complete
> > information including
> > properties.The complete output example is as follows:
> >
> > +-+---+
> > | catalog_description_item | catalog_description_value |
> >
> > +-+---+
> > | Name | cat1
> > |
> > | Type | generic_in_memory
> >|
> > | Comment |
> > |
> > | Properties |((k1,v1), (k2,v2))
> > |
> >
> > +-+---+
> >
> > 2. Could you add support for ALTER CATALOG xxx UNSET ('mykey')? This is
> > > also very useful in ALTER TABLE.
> >
> > I found that there is already an ALTER TABLE xxx RESET ('mykey') syntax [1]
> > now,
> > which will reset the myKey attribute of a certain table to the default
> > value. For catalogs,
> > it might be better to use ALTER CATALOG xxx RESET ('mykey') for the sake of
> > design
> > consistency.
> >
> > WDYT? Looking forward to your suggestions.
> >
> > Best,
> > Yubin
> >
> > [1]
> >
> > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/alter/#reset
> >
> >
> > On Mon, Mar 18, 2024 at 11:49 AM Jark Wu wrote:
> >
> > > Hi Yubin,
> > >
> > > Thanks for updating the FLIP. The updated version looks good in general.
> > > I only have 2 minor comments.
> > >
> > > 1. I think the purpose of DESCRIBE CATALOG is to display metadata
> > > information including catalog name,
> > > catalog comment (may be introduced in the future), catalog type, and
> > > catalog properties (for example [1]).
> > > Expanding all properties may limit this syntax to include more metadata
> > > information in the future.
> > >
> > > 2. Could you add support for ALTER CATALOG xxx UNSET ('mykey')? This is
> > > also very useful in ALTER TABLE.
> > >
> > > Best,
> > > Jark
> > >
> > > [1]:
> > >
> > >
> > https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-aux-describe-schema.html
> > >
> > >
> > >
> > > On Fri, 15 Mar 2024 at 12:06, Yubin Li wrote:
> > >
> > > > Hi Xuyang,
> > > >
> > > > Thank you for pointing this out, The parser part of `describe catalog`
> > > > syntax
> > > > has indeed been implemented in FLIP-69, and it is not actually
> > available.
> > > > we can complete the syntax in this FLIP [1]. I have updated the doc :)
> > > >
> > > > Best,
> > > > Yubin
> > > >
> > > > [1]
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-436%3A+Introduce+Catalog-related+Syntax
> > > >
> > > > On Fri, Mar 15, 2024 at 10:12 AM Xuyang wrote:
> > > >
> > > > > Hi, Yubin. Big +1 for this Flip. I just left one minor comment
> > > following.
> > > > >
> > > > >
> > > > > I found that although flink has not supported syntax 'DESCRIBE
> > CATALOG
> > > > > catalog_name' currently, it was already
> > > > > discussed in flip-69[1], do we need to restart discussing it?
> > > > > I don't have a particular preference regarding the restart
> > discussion.
> > > It
> > > > > seems that there is no difference on this syntax
> > > > > in FLIP-436, so maybe it would be best to refer back to FLIP-69 in
> > this
> > > > > FLIP. WDYT?
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-69%3A+Flink+SQL+DDL+Enhancement
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best！
> > > > > Xuyang
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > At 2024-03-15 02:49:59, "Yubin Li" wrote:
> > > > > >Hi folks,
> > > > > >
> > > > > >Thank you all for your input, it really makes sense to introduce
> > > missing
> > > > > >catalog-related SQL
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations! Best, Hang Paul Lam 于2024年3月18日周一 17:18写道： > Congrats! Thanks to everyone involved! > > Best, > Paul Lam > > > 2024年3月18日 16:37，Samrat Deb 写道： > > > > Congratulations ! > > > > On Mon, 18 Mar 2024 at 2:07 PM, Jingsong Li > wrote: > > > >> Congratulations! > >> > >> On Mon, Mar 18, 2024 at 4:30 PM Rui Fan <1996fan...@gmail.com> wrote: > >>> > >>> Congratulations, thanks for the great work! > >>> > >>> Best, > >>> Rui > >>> > >>> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee > >> wrote: > > The Apache Flink community is very happy to announce the release of > >> Apache Flink 1.19.0, which is the fisrt release for the Apache Flink > 1.19 > >> series. > > Apache Flink® is an open-source stream processing framework for > >> distributed, high-performing, always-available, and accurate data > streaming > >> applications. > > The release is available for download at: > https://flink.apache.org/downloads.html > > Please check out the release blog post for an overview of the > >> improvements for this bugfix release: > > >> > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > The full release notes are available in Jira: > > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > > We would like to thank all contributors of the Apache Flink community > >> who made this release possible! > > > Best, > Yun, Jing, Martijn and Lincoln > >> > >
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congrats! Thanks to everyone involved! Best, Paul Lam > 2024年3月18日 16:37，Samrat Deb 写道： > > Congratulations ! > > On Mon, 18 Mar 2024 at 2:07 PM, Jingsong Li wrote: > >> Congratulations! >> >> On Mon, Mar 18, 2024 at 4:30 PM Rui Fan <1996fan...@gmail.com> wrote: >>> >>> Congratulations, thanks for the great work! >>> >>> Best, >>> Rui >>> >>> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee >> wrote: The Apache Flink community is very happy to announce the release of >> Apache Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 >> series. Apache Flink® is an open-source stream processing framework for >> distributed, high-performing, always-available, and accurate data streaming >> applications. The release is available for download at: https://flink.apache.org/downloads.html Please check out the release blog post for an overview of the >> improvements for this bugfix release: >> https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ The full release notes are available in Jira: >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 We would like to thank all contributors of the Apache Flink community >> who made this release possible! Best, Yun, Jing, Martijn and Lincoln >>
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations! Thanks Lincoln, Yun, Martijn and Jing for driving this release. Thanks everyone involved. Best, Zakelly On Mon, Mar 18, 2024 at 5:05 PM weijie guo wrote: > Congratulations! > > Thanks release managers and all the contributors involved. > > Best regards, > > Weijie > > > Leonard Xu 于2024年3月18日周一 16:45写道： > >> Congratulations, thanks release managers and all involved for the great >> work! >> >> >> Best, >> Leonard >> >> > 2024年3月18日 下午4:32，Jingsong Li 写道： >> > >> > Congratulations! >> > >> > On Mon, Mar 18, 2024 at 4:30 PM Rui Fan <1996fan...@gmail.com> wrote: >> >> >> >> Congratulations, thanks for the great work! >> >> >> >> Best, >> >> Rui >> >> >> >> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee >> wrote: >> >>> >> >>> The Apache Flink community is very happy to announce the release of >> Apache Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 >> series. >> >>> >> >>> Apache Flink® is an open-source stream processing framework for >> distributed, high-performing, always-available, and accurate data streaming >> applications. >> >>> >> >>> The release is available for download at: >> >>> https://flink.apache.org/downloads.html >> >>> >> >>> Please check out the release blog post for an overview of the >> improvements for this bugfix release: >> >>> >> https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ >> >>> >> >>> The full release notes are available in Jira: >> >>> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 >> >>> >> >>> We would like to thank all contributors of the Apache Flink community >> who made this release possible! >> >>> >> >>> >> >>> Best, >> >>> Yun, Jing, Martijn and Lincoln >> >>
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations! Thanks release managers and all the contributors involved. Best regards, Weijie Leonard Xu 于2024年3月18日周一 16:45写道： > Congratulations, thanks release managers and all involved for the great > work! > > > Best, > Leonard > > > 2024年3月18日 下午4:32，Jingsong Li 写道： > > > > Congratulations! > > > > On Mon, Mar 18, 2024 at 4:30 PM Rui Fan <1996fan...@gmail.com> wrote: > >> > >> Congratulations, thanks for the great work! > >> > >> Best, > >> Rui > >> > >> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee > wrote: > >>> > >>> The Apache Flink community is very happy to announce the release of > Apache Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 > series. > >>> > >>> Apache Flink® is an open-source stream processing framework for > distributed, high-performing, always-available, and accurate data streaming > applications. > >>> > >>> The release is available for download at: > >>> https://flink.apache.org/downloads.html > >>> > >>> Please check out the release blog post for an overview of the > improvements for this bugfix release: > >>> > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > >>> > >>> The full release notes are available in Jira: > >>> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > >>> > >>> We would like to thank all contributors of the Apache Flink community > who made this release possible! > >>> > >>> > >>> Best, > >>> Yun, Jing, Martijn and Lincoln > >
Re: [VOTE] Apache Flink Kubernetes Operator Release 1.8.0, release candidate #1
Hi Max! +1 (binding) - Verified source release, helm chart + checkpoints / signatures - Helm points to correct image - Deployed operator, stateful example and executed upgrade + savepoint redeploy - Verified logs - Flink web PR looks good +1 A minor correction is that [3] in the email should point to: ghcr.io/apache/flink-kubernetes-operator:91d67d9 . But the helm chart and everything is correct. It's a typo in the vote email. Thank you for preparing the release! Cheers, Gyula On Mon, Mar 18, 2024 at 8:26 AM Rui Fan <1996fan...@gmail.com> wrote: > Thanks Max for driving this release! > > +1(non-binding) > > - Downloaded artifacts from dist ( svn co > > https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.8.0-rc1/ > ) > - Verified SHA512 checksums : ( for i in *.tgz; do echo $i; sha512sum > --check $i.sha512; done ) > - Verified GPG signatures : ( $ for i in *.tgz; do echo $i; gpg --verify > $i.asc $i ) > - Build the source with java-11 and java-17 ( mvn -T 20 clean install > -DskipTests ) > - Verified the license header during build the source > - Verified that chart and appVersion matches the target release (less the > index.yaml and Chart.yaml ) > - RC repo works as Helm repo( helm repo add flink-operator-repo-1.8.0-rc1 > > https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.8.0-rc1/ > ) > - Verified Helm chart can be installed ( helm install > flink-kubernetes-operator > flink-operator-repo-1.8.0-rc1/flink-kubernetes-operator --set > webhook.create=false ) > - Submitted the autoscaling demo, the autoscaler works well with *memory > tuning *(kubectl apply -f autoscaling.yaml) >- job.autoscaler.memory.tuning.enabled: "true" > - Download Autoscaler standalone: wget > > https://repository.apache.org/content/repositories/orgapacheflink-1710/org/apache/flink/flink-autoscaler-standalone/1.8.0/flink-autoscaler-standalone-1.8.0.jar > - Ran Autoscaler standalone locally, it works well with rescale api and > JDBC state store/event handler > > Best, > Rui > > On Fri, Mar 15, 2024 at 1:45 AM Maximilian Michels wrote: > > > Hi everyone, > > > > Please review and vote on the release candidate #1 for the version > > 1.8.0 of the Apache Flink Kubernetes Operator, as follows: > > > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > **Release Overview** > > > > As an overview, the release consists of the following: > > a) Kubernetes Operator canonical source distribution (including the > > Dockerfile), to be deployed to the release repository at > > dist.apache.org > > b) Kubernetes Operator Helm Chart to be deployed to the release > > repository at dist.apache.org > > c) Maven artifacts to be deployed to the Maven Central Repository > > d) Docker image to be pushed to Dockerhub > > > > **Staging Areas to Review** > > > > The staging areas containing the above mentioned artifacts are as > > follows, for your review: > > * All artifacts for (a), (b) can be found in the corresponding dev > > repository at dist.apache.org [1] > > * All artifacts for (c) can be found at the Apache Nexus Repository [2] > > * The docker image for (d) is staged on github [3] > > > > All artifacts are signed with the key > > DA359CBFCEB13FC302A8793FB655E6F7693D5FDE [4] > > > > Other links for your review: > > * JIRA release notes [5] > > * source code tag "release-1.8.0-rc1" [6] > > * PR to update the website Downloads page to include Kubernetes > > Operator links [7] > > > > **Vote Duration** > > > > The voting time will run for at least 72 hours. It is adopted by > > majority approval, with at least 3 PMC affirmative votes. > > > > **Note on Verification** > > > > You can follow the basic verification guide here [8]. Note that you > > don't need to verify everything yourself, but please make note of what > > you have tested together with your +- vote. > > > > Thanks, > > Max > > > > [1] > > > https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.8.0-rc1/ > > [2] > > https://repository.apache.org/content/repositories/orgapacheflink-1710/ > > [3] ghcr.io/apache/flink-kubernetes-operator:8938658 > > [4] https://dist.apache.org/repos/dist/release/flink/KEYS > > [5] > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12353866=12315522 > > [6] > > > https://github.com/apache/flink-kubernetes-operator/tree/release-1.8.0-rc1 > > [7] https://github.com/apache/flink-web/pull/726 > > [8] > > > https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Kubernetes+Operator+Release > > >
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations, thanks release managers and all involved for the great work! Best, Leonard > 2024年3月18日 下午4:32，Jingsong Li 写道： > > Congratulations! > > On Mon, Mar 18, 2024 at 4:30 PM Rui Fan <1996fan...@gmail.com> wrote: >> >> Congratulations, thanks for the great work! >> >> Best, >> Rui >> >> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee wrote: >>> >>> The Apache Flink community is very happy to announce the release of Apache >>> Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 series. >>> >>> Apache Flink® is an open-source stream processing framework for >>> distributed, high-performing, always-available, and accurate data streaming >>> applications. >>> >>> The release is available for download at: >>> https://flink.apache.org/downloads.html >>> >>> Please check out the release blog post for an overview of the improvements >>> for this bugfix release: >>> https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ >>> >>> The full release notes are available in Jira: >>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 >>> >>> We would like to thank all contributors of the Apache Flink community who >>> made this release possible! >>> >>> >>> Best, >>> Yun, Jing, Martijn and Lincoln
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations ! On Mon, 18 Mar 2024 at 2:07 PM, Jingsong Li wrote: > Congratulations! > > On Mon, Mar 18, 2024 at 4:30 PM Rui Fan <1996fan...@gmail.com> wrote: > > > > Congratulations, thanks for the great work! > > > > Best, > > Rui > > > > On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee > wrote: > >> > >> The Apache Flink community is very happy to announce the release of > Apache Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 > series. > >> > >> Apache Flink® is an open-source stream processing framework for > distributed, high-performing, always-available, and accurate data streaming > applications. > >> > >> The release is available for download at: > >> https://flink.apache.org/downloads.html > >> > >> Please check out the release blog post for an overview of the > improvements for this bugfix release: > >> > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > >> > >> The full release notes are available in Jira: > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > >> > >> We would like to thank all contributors of the Apache Flink community > who made this release possible! > >> > >> > >> Best, > >> Yun, Jing, Martijn and Lincoln >
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congrats! Thanks Lincoln, Jing, Yun and Martijn driving this release. Thanks all who involved this release! Best, Jark On Mon, 18 Mar 2024 at 16:31, Rui Fan <1996fan...@gmail.com> wrote: > Congratulations, thanks for the great work! > > Best, > Rui > > On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee > wrote: > > > The Apache Flink community is very happy to announce the release of > Apache > > Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 > series. > > > > Apache Flink® is an open-source stream processing framework for > > distributed, high-performing, always-available, and accurate data > streaming > > applications. > > > > The release is available for download at: > > https://flink.apache.org/downloads.html > > > > Please check out the release blog post for an overview of the > improvements > > for this bugfix release: > > > > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > > > The full release notes are available in Jira: > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > > > > We would like to thank all contributors of the Apache Flink community who > > made this release possible! > > > > > > Best, > > Yun, Jing, Martijn and Lincoln > > >
Re: Re: [DISCUSS] FLIP-436: Introduce "SHOW CREATE CATALOG" Syntax
Hi Yubin,
Thanks for the quick response. The suggestion sounds good to me!
Best,
Jark
On Mon, 18 Mar 2024 at 13:06, Yubin Li wrote:
> Hi Jark,
>
> Good pointing! Thanks for your reply, there are some details to align :)
>
> 1. I think the purpose of DESCRIBE CATALOG is to display metadata
> > information including catalog name,
> > catalog comment (may be introduced in the future), catalog type, and
> > catalog properties (for example [1])
>
> Adopting { DESC | DESCRIBE } CATALOG [ EXTENDED ] xx as formal syntax,
> Producing rich and compatible results for future needs is very important.
> When
> specifying "extended" in the syntax, it will output the complete
> information including
> properties.The complete output example is as follows:
>
> +-+---+
> | catalog_description_item | catalog_description_value |
>
> +-+---+
> | Name | cat1
> |
> | Type | generic_in_memory
>|
> | Comment |
> |
> | Properties |((k1,v1), (k2,v2))
> |
>
> +-+---+
>
> 2. Could you add support for ALTER CATALOG xxx UNSET ('mykey')? This is
> > also very useful in ALTER TABLE.
>
> I found that there is already an ALTER TABLE xxx RESET ('mykey') syntax [1]
> now,
> which will reset the myKey attribute of a certain table to the default
> value. For catalogs,
> it might be better to use ALTER CATALOG xxx RESET ('mykey') for the sake of
> design
> consistency.
>
> WDYT? Looking forward to your suggestions.
>
> Best,
> Yubin
>
> [1]
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/alter/#reset
>
>
> On Mon, Mar 18, 2024 at 11:49 AM Jark Wu wrote:
>
> > Hi Yubin,
> >
> > Thanks for updating the FLIP. The updated version looks good in general.
> > I only have 2 minor comments.
> >
> > 1. I think the purpose of DESCRIBE CATALOG is to display metadata
> > information including catalog name,
> > catalog comment (may be introduced in the future), catalog type, and
> > catalog properties (for example [1]).
> > Expanding all properties may limit this syntax to include more metadata
> > information in the future.
> >
> > 2. Could you add support for ALTER CATALOG xxx UNSET ('mykey')? This is
> > also very useful in ALTER TABLE.
> >
> > Best,
> > Jark
> >
> > [1]:
> >
> >
> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-aux-describe-schema.html
> >
> >
> >
> > On Fri, 15 Mar 2024 at 12:06, Yubin Li wrote:
> >
> > > Hi Xuyang,
> > >
> > > Thank you for pointing this out, The parser part of `describe catalog`
> > > syntax
> > > has indeed been implemented in FLIP-69, and it is not actually
> available.
> > > we can complete the syntax in this FLIP [1]. I have updated the doc :)
> > >
> > > Best,
> > > Yubin
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-436%3A+Introduce+Catalog-related+Syntax
> > >
> > > On Fri, Mar 15, 2024 at 10:12 AM Xuyang wrote:
> > >
> > > > Hi, Yubin. Big +1 for this Flip. I just left one minor comment
> > following.
> > > >
> > > >
> > > > I found that although flink has not supported syntax 'DESCRIBE
> CATALOG
> > > > catalog_name' currently, it was already
> > > > discussed in flip-69[1], do we need to restart discussing it?
> > > > I don't have a particular preference regarding the restart
> discussion.
> > It
> > > > seems that there is no difference on this syntax
> > > > in FLIP-436, so maybe it would be best to refer back to FLIP-69 in
> this
> > > > FLIP. WDYT?
> > > >
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-69%3A+Flink+SQL+DDL+Enhancement
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best！
> > > > Xuyang
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > At 2024-03-15 02:49:59, "Yubin Li" wrote:
> > > > >Hi folks,
> > > > >
> > > > >Thank you all for your input, it really makes sense to introduce
> > missing
> > > > >catalog-related SQL syntaxes under this FLIP, and I have changed the
> > > > >title of doc to "FLIP-436: Introduce Catalog-related Syntax".
> > > > >
> > > > >After comprehensive consideration, the following syntaxes should be
> > > > >introduced, more suggestions are welcome :)
> > > > >
> > > > >> 1. SHOW CREATE CATALOG catalog_name
> > > > >> 2. DESCRIBE/DESC CATALOG catalog_name
> > > > >> 3. ALTER CATALOG catalog_name SET (key1=val1, key2=val2, ...)
> > > > >
> > > > >Regarding the `alter catalog` syntax format, I refer to the current
> > > design
> > > > >of `alter database`.
> > > > >
> > > > >Given that CatalogManager already provides catalog operations such
> as
>
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations! On Mon, Mar 18, 2024 at 4:30 PM Rui Fan <1996fan...@gmail.com> wrote: > > Congratulations, thanks for the great work! > > Best, > Rui > > On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee wrote: >> >> The Apache Flink community is very happy to announce the release of Apache >> Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 series. >> >> Apache Flink® is an open-source stream processing framework for distributed, >> high-performing, always-available, and accurate data streaming applications. >> >> The release is available for download at: >> https://flink.apache.org/downloads.html >> >> Please check out the release blog post for an overview of the improvements >> for this bugfix release: >> https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ >> >> The full release notes are available in Jira: >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 >> >> We would like to thank all contributors of the Apache Flink community who >> made this release possible! >> >> >> Best, >> Yun, Jing, Martijn and Lincoln
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations, thanks for the great work! Best, Rui On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 series. > > Apache Flink® is an open-source stream processing framework for > distributed, high-performing, always-available, and accurate data streaming > applications. > > The release is available for download at: > https://flink.apache.org/downloads.html > > Please check out the release blog post for an overview of the improvements > for this bugfix release: > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > The full release notes are available in Jira: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > > We would like to thank all contributors of the Apache Flink community who > made this release possible! > > > Best, > Yun, Jing, Martijn and Lincoln >
[ANNOUNCE] Apache Flink 1.19.0 released
The Apache Flink community is very happy to announce the release of Apache Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for download at: https://flink.apache.org/downloads.html Please check out the release blog post for an overview of the improvements for this bugfix release: https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ The full release notes are available in Jira: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 We would like to thank all contributors of the Apache Flink community who made this release possible! Best, Yun, Jing, Martijn and Lincoln
[jira] [Created] (FLINK-34715) Fix mysql ut about closing BinlogSplitReader
Xin Gong created FLINK-34715: Summary: Fix mysql ut about closing BinlogSplitReader Key: FLINK-34715 URL: https://issues.apache.org/jira/browse/FLINK-34715 Project: Flink Issue Type: Improvement Components: Flink CDC Reporter: Xin Gong Fix For: cdc-3.1.0 BinlogSplitReaderTest#readBinlogSplitsFromSnapshotSplits should test binlog reader is closed after binlog reader close. But code always test snapshot split reader is closed. ```java binlogReader.close(); assertNotNull(snapshotSplitReader.getExecutorService()); assertTrue(snapshotSplitReader.getExecutorService().isTerminated()); ``` We shoud change code to ```java binlogReader.close(); assertNotNull(binlogReader.getExecutorService()); assertTrue(binlogReader.getExecutorService().isTerminated()); ``` -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] Apache Flink Kubernetes Operator Release 1.8.0, release candidate #1
Thanks Max for driving this release! +1(non-binding) - Downloaded artifacts from dist ( svn co https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.8.0-rc1/ ) - Verified SHA512 checksums : ( for i in *.tgz; do echo $i; sha512sum --check $i.sha512; done ) - Verified GPG signatures : ( $ for i in *.tgz; do echo $i; gpg --verify $i.asc $i ) - Build the source with java-11 and java-17 ( mvn -T 20 clean install -DskipTests ) - Verified the license header during build the source - Verified that chart and appVersion matches the target release (less the index.yaml and Chart.yaml ) - RC repo works as Helm repo( helm repo add flink-operator-repo-1.8.0-rc1 https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.8.0-rc1/ ) - Verified Helm chart can be installed ( helm install flink-kubernetes-operator flink-operator-repo-1.8.0-rc1/flink-kubernetes-operator --set webhook.create=false ) - Submitted the autoscaling demo, the autoscaler works well with *memory tuning *(kubectl apply -f autoscaling.yaml) - job.autoscaler.memory.tuning.enabled: "true" - Download Autoscaler standalone: wget https://repository.apache.org/content/repositories/orgapacheflink-1710/org/apache/flink/flink-autoscaler-standalone/1.8.0/flink-autoscaler-standalone-1.8.0.jar - Ran Autoscaler standalone locally, it works well with rescale api and JDBC state store/event handler Best, Rui On Fri, Mar 15, 2024 at 1:45 AM Maximilian Michels wrote: > Hi everyone, > > Please review and vote on the release candidate #1 for the version > 1.8.0 of the Apache Flink Kubernetes Operator, as follows: > > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > **Release Overview** > > As an overview, the release consists of the following: > a) Kubernetes Operator canonical source distribution (including the > Dockerfile), to be deployed to the release repository at > dist.apache.org > b) Kubernetes Operator Helm Chart to be deployed to the release > repository at dist.apache.org > c) Maven artifacts to be deployed to the Maven Central Repository > d) Docker image to be pushed to Dockerhub > > **Staging Areas to Review** > > The staging areas containing the above mentioned artifacts are as > follows, for your review: > * All artifacts for (a), (b) can be found in the corresponding dev > repository at dist.apache.org [1] > * All artifacts for (c) can be found at the Apache Nexus Repository [2] > * The docker image for (d) is staged on github [3] > > All artifacts are signed with the key > DA359CBFCEB13FC302A8793FB655E6F7693D5FDE [4] > > Other links for your review: > * JIRA release notes [5] > * source code tag "release-1.8.0-rc1" [6] > * PR to update the website Downloads page to include Kubernetes > Operator links [7] > > **Vote Duration** > > The voting time will run for at least 72 hours. It is adopted by > majority approval, with at least 3 PMC affirmative votes. > > **Note on Verification** > > You can follow the basic verification guide here [8]. Note that you > don't need to verify everything yourself, but please make note of what > you have tested together with your +- vote. > > Thanks, > Max > > [1] > https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.8.0-rc1/ > [2] > https://repository.apache.org/content/repositories/orgapacheflink-1710/ > [3] ghcr.io/apache/flink-kubernetes-operator:8938658 > [4] https://dist.apache.org/repos/dist/release/flink/KEYS > [5] > https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12353866=12315522 > [6] > https://github.com/apache/flink-kubernetes-operator/tree/release-1.8.0-rc1 > [7] https://github.com/apache/flink-web/pull/726 > [8] > https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Kubernetes+Operator+Release >
[jira] [Created] (FLINK-34713) CLONE - Updates the docs stable version
lincoln lee created FLINK-34713:
---
Summary: CLONE - Updates the docs stable version
Key: FLINK-34713
URL: https://issues.apache.org/jira/browse/FLINK-34713
Project: Flink
Issue Type: Sub-task
Reporter: Sergey Nuyanzin
Update docs to "stable" in {{docs/config.toml}} in the branch of the
_just-released_ version:
* Change V{{{}ersion{}}} from {{{}x.y-SNAPSHOT }}to \{{{}x.y.z{}}}, i.e.
{{1.6-SNAPSHOT}} to {{1.6.0}}
* Change V{{{}ersionTitle{}}} from {{x.y-SNAPSHOT}} to {{{}x.y{}}}, i.e.
{{1.6-SNAPSHOT}} to {{1.6}}
* Change Branch from {{master}} to {{{}release-x.y{}}}, i.e. {{master}} to
{{release-1.6}}
* Change {{baseURL}} from
{{//[ci.apache.org/projects/flink/flink-docs-master|http://ci.apache.org/projects/flink/flink-docs-master]}}
to
{{//[ci.apache.org/projects/flink/flink-docs-release-x.y|http://ci.apache.org/projects/flink/flink-docs-release-x.y]}}
* Change {{javadocs_baseurl}} from
{{//[ci.apache.org/projects/flink/flink-docs-master|http://ci.apache.org/projects/flink/flink-docs-master]}}
to
{{//[ci.apache.org/projects/flink/flink-docs-release-x.y|http://ci.apache.org/projects/flink/flink-docs-release-x.y]}}
* Change {{IsStable}} to {{true}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34712) CLONE - Update reference data for Migration Tests
lincoln lee created FLINK-34712:
---
Summary: CLONE - Update reference data for Migration Tests
Key: FLINK-34712
URL: https://issues.apache.org/jira/browse/FLINK-34712
Project: Flink
Issue Type: Sub-task
Reporter: Sergey Nuyanzin
Assignee: Sergey Nuyanzin
Fix For: 1.19.0, 1.18.1
Update migration tests in master to cover migration from new version. Since
1.18, this step could be done automatically with the following steps. For more
information please refer to [this
page.|https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-migration-test-utils/README.md]
# {*}On the published release tag (e.g., release-1.16.0){*}, run
{panel}
{panel}
|{{$ mvn clean }}{{package}} {{{}-Pgenerate-migration-test-data
-Dgenerate.version={}}}{{{}1.16{}}} {{-nsu -Dfast -DskipTests}}|
The version (1.16 in the command above) should be replaced with the target one.
# Modify the content of the file
[apache/flink:flink-test-utils-parent/flink-migration-test-utils/src/main/resources/most_recently_published_version|https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-migration-test-utils/src/main/resources/most_recently_published_version]
to the latest version (it would be "v1_16" if sticking to the example where
1.16.0 was released).
# Commit the modification in step a and b with "{_}[release] Generate
reference data for state migration tests based on release-1.xx.0{_}" to the
corresponding release branch (e.g. {{release-1.16}} in our example), replace
"xx" with the actual version (in this example "16"). You should use the Jira
issue ID in case of [release] as the commit message's prefix if you have a
dedicated Jira issue for this task.
# Cherry-pick the commit to the master branch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34709) CLONE - Remove outdated versions
lincoln lee created FLINK-34709:
---
Summary: CLONE - Remove outdated versions
Key: FLINK-34709
URL: https://issues.apache.org/jira/browse/FLINK-34709
Project: Flink
Issue Type: Sub-task
Reporter: Sergey Nuyanzin
h4. dist.apache.org
For a new major release remove all release files older than 2 versions, e.g.,
when releasing 1.7, remove all releases <= 1.5.
For a new bugfix version remove all release files for previous bugfix releases
in the same series, e.g., when releasing 1.7.1, remove the 1.7.0 release.
# If you have not already, check out the Flink section of the {{release}}
repository on {{[dist.apache.org|http://dist.apache.org/]}} via Subversion. In
a fresh directory:
{code}
svn checkout https://dist.apache.org/repos/dist/release/flink --depth=immediates
cd flink
{code}
# Remove files for outdated releases and commit the changes.
{code}
svn remove flink-
svn commit
{code}
# Verify that files are
[removed|https://dist.apache.org/repos/dist/release/flink]
(!) Remember to remove the corresponding download links from the website.
h4. CI
Disable the cron job for the now-unsupported version from
(tools/azure-pipelines/[build-apache-repo.yml|https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-apache-repo.yml])
in the respective branch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34714) CLONE - Start End of Life discussion thread for now outdated Flink minor version
lincoln lee created FLINK-34714: --- Summary: CLONE - Start End of Life discussion thread for now outdated Flink minor version Key: FLINK-34714 URL: https://issues.apache.org/jira/browse/FLINK-34714 Project: Flink Issue Type: Sub-task Reporter: Matthias Pohl The idea is to discuss whether we should do a final release for the now not supported minor version in the community. Such a minor release shouldn't be covered by the current minor version release managers. Their only responsibility is to trigger the discussion. The intention of a final patch release for the now unsupported Flink minor version is to flush out all the fixes that didn't end up in the previous release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34711) CLONE - Other announcements
lincoln lee created FLINK-34711:
---
Summary: CLONE - Other announcements
Key: FLINK-34711
URL: https://issues.apache.org/jira/browse/FLINK-34711
Project: Flink
Issue Type: Sub-task
Reporter: Sergey Nuyanzin
h3. Recordkeeping
Use [reporter.apache.org|https://reporter.apache.org/addrelease.html?flink] to
seed the information about the release into future project reports.
(Note: Only PMC members have access report releases. If you do not have access,
ask on the mailing list for assistance.)
h3. Flink blog
Major or otherwise important releases should have a blog post. Write one if
needed for this particular release. Minor releases that don’t introduce new
major functionality don’t necessarily need to be blogged (see [flink-web PR
#581 for Flink 1.15.3|https://github.com/apache/flink-web/pull/581] as an
example for a minor release blog post).
Please make sure that the release notes of the documentation (see section
"Review and update documentation") are linked from the blog post of a major
release.
We usually include the names of all contributors in the announcement blog post.
Use the following command to get the list of contributors:
{code}
# first line is required to make sort first with uppercase and then lower
export LC_ALL=C
export FLINK_PREVIOUS_RELEASE_BRANCH=
export FLINK_CURRENT_RELEASE_BRANCH=
# e.g.
# export FLINK_PREVIOUS_RELEASE_BRANCH=release-1.17
# export FLINK_CURRENT_RELEASE_BRANCH=release-1.18
git log $(git merge-base master $FLINK_PREVIOUS_RELEASE_BRANCH)..$(git show-ref
--hash ${FLINK_CURRENT_RELEASE_BRANCH}) --pretty=format:"%an%n%cn" | sort -u |
paste -sd, | sed "s/\,/\, /g"
{code}
h3. Social media
Tweet, post on Facebook, LinkedIn, and other platforms. Ask other contributors
to do the same.
h3. Flink Release Wiki page
Add a summary of things that went well or that went not so well during the
release process. This can include feedback from contributors but also more
generic things like the release have taken longer than initially anticipated
(and why) to give a bit of context to the release process.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34710) CLONE - Apache mailing lists announcements
lincoln lee created FLINK-34710:
---
Summary: CLONE - Apache mailing lists announcements
Key: FLINK-34710
URL: https://issues.apache.org/jira/browse/FLINK-34710
Project: Flink
Issue Type: Sub-task
Reporter: Sergey Nuyanzin
Announce on the {{dev@}} mailing list that the release has been finished.
Announce on the release on the {{user@}} mailing list, listing major
improvements and contributions.
Announce the release on the [annou...@apache.org|mailto:annou...@apache.org]
mailing list.
{panel}
{panel}
|{{From: Release Manager}}
{{To: dev@flink.apache.org, u...@flink.apache.org, user...@flink.apache.org,
annou...@apache.org}}
{{Subject: [ANNOUNCE] Apache Flink 1.2.3 released}}
{{The Apache Flink community is very happy to announce the release of Apache
Flink 1.2.3, which is the third bugfix release for the Apache Flink 1.2
series.}}
{{Apache Flink® is an open-source stream processing framework for distributed,
high-performing, always-available, and accurate data streaming applications.}}
{{The release is available for download at:}}
{{[https://flink.apache.org/downloads.html]}}
{{Please check out the release blog post for an overview of the improvements
for this bugfix release:}}
{{}}
{{The full release notes are available in Jira:}}
{{}}
{{We would like to thank all contributors of the Apache Flink community who
made this release possible!}}
{{Feel free to reach out to the release managers (or respond to this thread)
with feedback on the release process. Our goal is to constantly improve the
release process. Feedback on what could be improved or things that didn't go so
well are appreciated.}}
{{Regards,}}
{{Release Manager}}|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34708) CLONE - Merge website pull request
lincoln lee created FLINK-34708: --- Summary: CLONE - Merge website pull request Key: FLINK-34708 URL: https://issues.apache.org/jira/browse/FLINK-34708 Project: Flink Issue Type: Sub-task Reporter: Sergey Nuyanzin Merge the website pull request to [list the release|http://flink.apache.org/downloads.html]. Make sure to regenerate the website as well, as it isn't build automatically. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34707) CLONE - Update japicmp configuration
lincoln lee created FLINK-34707:
---
Summary: CLONE - Update japicmp configuration
Key: FLINK-34707
URL: https://issues.apache.org/jira/browse/FLINK-34707
Project: Flink
Issue Type: Sub-task
Reporter: Sergey Nuyanzin
Assignee: Sergey Nuyanzin
Fix For: 1.19.0, 1.18.1
Update the japicmp reference version and wipe exclusions / enable API
compatibility checks for {{@PublicEvolving}} APIs on the corresponding SNAPSHOT
branch with the {{update_japicmp_configuration.sh}} script (see below).
For a new major release (x.y.0), run the same command also on the master branch
for updating the japicmp reference version and removing out-dated exclusions in
the japicmp configuration.
Make sure that all Maven artifacts are already pushed to Maven Central.
Otherwise, there's a risk that CI fails due to missing reference artifacts.
{code:bash}
tools $ NEW_VERSION=$RELEASE_VERSION releasing/update_japicmp_configuration.sh
tools $ cd ..$ git add *$ git commit -m "Update japicmp configuration for
$RELEASE_VERSION" {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34706) Promote release 1.19
lincoln lee created FLINK-34706:
---
Summary: Promote release 1.19
Key: FLINK-34706
URL: https://issues.apache.org/jira/browse/FLINK-34706
Project: Flink
Issue Type: New Feature
Affects Versions: 1.18.0
Reporter: Sergey Nuyanzin
Assignee: Jing Ge
Once the release has been finalized (FLINK-32920), the last step of the process
is to promote the release within the project and beyond. Please wait for 24h
after finalizing the release in accordance with the [ASF release
policy|http://www.apache.org/legal/release-policy.html#release-announcements].
*Final checklist to declare this issue resolved:*
# Website pull request to [list the
release|http://flink.apache.org/downloads.html] merged
# Release announced on the user@ mailing list.
# Blog post published, if applicable.
# Release recorded in
[reporter.apache.org|https://reporter.apache.org/addrelease.html?flink].
# Release announced on social media.
# Completion declared on the dev@ mailing list.
# Update Homebrew: [https://docs.brew.sh/How-To-Open-a-Homebrew-Pull-Request]
(seems to be done automatically - at least for minor releases for both minor
and major releases)
# Updated the japicmp configuration
** corresponding SNAPSHOT branch japicmp reference version set to the just
released version, and API compatibiltity checks for {{@PublicEvolving}} was
enabled
** (minor version release only) master branch japicmp reference version set to
the just released version
** (minor version release only) master branch japicmp exclusions have been
cleared
# Update the list of previous version in {{docs/config.toml}} on the master
branch.
# Set {{show_outdated_warning: true}} in {{docs/config.toml}} in the branch of
the _now deprecated_ Flink version (i.e. 1.16 if 1.18.0 is released)
# Update stable and master alias in
[https://github.com/apache/flink/blob/master/.github/workflows/docs.yml]
# Open discussion thread for End of Life for Unsupported version (i.e. 1.16)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Apache Flink : Vector database connector
Hey, Asimansu Happy to hear you’re interested to integrate vector database and Apache Flink, I’ve talked similar topic with milvus community[1] members, I am also very familiar with the CDC, thus I know the underlying value well that the integration of the two can bring to AI users. If you are willing to contribute the corresponding connector, I will be very happy to take time to help review FLIP[2] and code in the community. Best, Leonard [1] https://github.com/milvus-io/milvus [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP+Connector+Template > 2024年3月17日 上午2:18，Asimansu Bera 写道： > > Hello Dev, > > Vector databases like Waveiate are utilized in GenAI-based applications to > store vector data in numeric format, supporting CRUD operations. > Write-ahead logging (WAL) is employed to capture any changes made to the > vector databases. For streaming-based applications, it's uncertain if > tracking changes in vector classes and objects is significant. However, > incorporating CDC (Change Data Capture) connectors can enhance processing > capabilities for vector data in streaming applications, especially for > LLM-based applications focused on data ingestion. > > https://lnkd.in/gByTZYDY > > Thoughts? > > Note: all RDBMS are introducing support for vector data.
