pnowojski commented on a change in pull request #352: URL: https://github.com/apache/flink-web/pull/352#discussion_r446008533
########## File path: _posts/2020-06-29-release-1.11.0.md ########## @@ -0,0 +1,299 @@ +--- +layout: post +title: "Apache Flink 1.11.0 Release Announcement" +date: 2020-06-29T08:00:00.000Z +categories: news +authors: +- morsapaes: + name: "Marta Paes" + twitter: "morsapaes" + +excerpt: The Apache Flink community is proud to announce the release of Flink 1.11.0! This release marks the first milestone in realizing a new vision for fault tolerance in Flink, and adds a handful of new features that simplify (and unify) Flink handling across the API stack. In particular for users of the Table API/SQL, this release introduces significant improvements to usability and opens up completely new use cases, including the much-anticipated support for Change Data Capture (CDC)! A great deal of effort has also gone into optimizing PyFlink and ensuring that its functionality is available to a broader set of Flink users. +--- + +The Apache Flink community is proud to announce the release of Flink 1.11.0! This release marks the first milestone in realizing a new vision for fault tolerance in Flink, and adds a handful of new features that simplify (and unify) Flink handling across the API stack. In particular for users of the Table API/SQL, this release introduces significant improvements to usability and opens up completely new use cases, including the much-anticipated support for Change Data Capture (CDC)! A great deal of effort has also gone into optimizing PyFlink and ensuring that its functionality is available to a broader set of Flink users. + +This blog post describes all major new features and improvements, important changes to be aware of and what to expect moving forward. + +{% toc %} + +The binary distribution and source artifacts are now available on the updated [Downloads page]({{ site.baseurl }}/downloads.html) of the Flink website, and the most recent distribution of PyFlink is available on [PyPI](https://pypi.org/project/apache-flink/). For more details, check the complete [release changelog](https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12346364&styleName=Html&projectId=12315522) and the [updated documentation]({{ site.DOCS_BASE_URL }}flink-docs-release-1.11/flink-docs-release-1.11/). + +We encourage you to download the release and share your feedback with the community through the [Flink mailing lists](https://flink.apache.org/community.html#mailing-lists) or [JIRA](https://issues.apache.org/jira/projects/FLINK/summary). + +## New Features and Improvements + +### Unaligned Checkpoints (Beta) + +Triggering a checkpoint in Flink will cause a [checkpoint barrier]({{ site.DOCS_BASE_URL }}flink-docs-release-1.11/internals/stream_checkpointing.html#barriers) to flow from the sources of your topology all the way towards the sinks. For operators that receive more than one input stream, the barriers flowing through each channel need to be aligned before the operator can snapshot its state and forward the checkpoint barrier — typically, this alignment will take just a few milliseconds to complete, but it can become a bottleneck in backpressured pipelines as: + + * Checkpoint barriers will flow much slower through backpressured channels, effectively blocking the remaining channels and their upstream operators during checkpointing; + + * Slow checkpoint barrier propagation leads to longer checkpointing times and can, worst case, result in little to no progress in the application. + +To improve the performance of checkpointing under backpressure scenarios, the community is rolling out the first iteration of unaligned checkpoints ([FLIP-76](https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints)) with Flink 1.11. Compared to the original checkpointing mechanism (Fig. 1), this approach doesn’t wait for barrier alignment across input channels, instead allowing barriers to overtake in-flight records and forwarding them downstream before the synchronous part of the checkpoint takes place (Fig. 2). + +<div style="line-height:60%;"> + <br> +</div> + +<div class="row"> + <div class="col-lg-6"> + <div class="text-center"> + <figure> + <img src="{{ site.baseurl }}/img/blog/2020-06-29-release-1.11.0/image1.gif" width="600px" alt="Aligned Checkpoints"/> + <br/><br/> + <figcaption><i><b>Fig.1:</b> Aligned Checkpoints</i></figcaption> + </figure> + </div> + </div> + <div class="col-lg-6"> + <div class="text-center"> + <figure> + <img src="{{ site.baseurl }}/img/blog/2020-06-29-release-1.11.0/image2.png" width="600px" alt="Unaligned Checkpoints"/> + <br/><br/> + <figcaption><i><b>Fig.2:</b> Unaligned Checkpoints</i></figcaption> + </figure> + </div> + </div> +</div> + +<div style="line-height:150%;"> + <br> +</div> + +Because in-flight records have to be persisted as part of the snapshot, unaligned checkpoints will lead to increased checkpoints sizes. On the upside, **checkpointing times are heavily reduced**, so users will see more progress (even in unstable environments) as more up-to-date checkpoints will lighten the recovery process. You can learn more about the current limitations of unaligned checkpoints in the [documentation]({{ site.DOCS_BASE_URL }}flink-docs-release-1.11/concepts/stateful-stream-processing.html#unaligned-checkpointing), and track the improvement work planned for this feature in [FLINK-14551](https://issues.apache.org/jira/browse/FLINK-14551). Review comment: It is documented https://github.com/apache/flink/pull/12722 , but I'm fine if you, @morsapaes and @AHeise think it's worth mentioning here as well. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org