This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
The following commit(s) were added to refs/heads/asf-site by this push:
new ffa580a1b Rebuild website
ffa580a1b is described below
commit ffa580a1b5e893e3360ec7a06c445a210b77e503
Author: MartijnVisser <[email protected]>
AuthorDate: Mon May 23 10:06:52 2022 +0200
Rebuild website
---
content/2022/05/23/latency-part2.html | 356 ++++++++++++++++
content/blog/feed.xml | 448 ++++-----------------
content/blog/index.html | 36 +-
content/blog/page10/index.html | 36 +-
content/blog/page11/index.html | 36 +-
content/blog/page12/index.html | 38 +-
content/blog/page13/index.html | 38 +-
content/blog/page14/index.html | 38 +-
content/blog/page15/index.html | 40 +-
content/blog/page16/index.html | 40 +-
content/blog/page17/index.html | 40 +-
content/blog/page18/index.html | 40 +-
content/blog/page19/index.html | 25 ++
content/blog/page2/index.html | 36 +-
content/blog/page3/index.html | 38 +-
content/blog/page4/index.html | 40 +-
content/blog/page5/index.html | 38 +-
content/blog/page6/index.html | 38 +-
content/blog/page7/index.html | 41 +-
content/blog/page8/index.html | 39 +-
content/blog/page9/index.html | 36 +-
.../img/blog/2022-05-23-latency-part2/async-io.png | Bin 0 -> 110500 bytes
.../enriching-with-async-io.png | Bin 0 -> 246599 bytes
.../blog/2022-05-23-latency-part2/spread-work.png | Bin 0 -> 153305 bytes
content/index.html | 6 +-
content/zh/index.html | 6 +-
26 files changed, 908 insertions(+), 621 deletions(-)
diff --git a/content/2022/05/23/latency-part2.html
b/content/2022/05/23/latency-part2.html
new file mode 100644
index 000000000..b417caeb1
--- /dev/null
+++ b/content/2022/05/23/latency-part2.html
@@ -0,0 +1,356 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <!-- The above 3 meta tags *must* come first in the head; any other head
content must come *after* these tags -->
+ <title>Apache Flink: Getting into Low-Latency Gears with Apache Flink -
Part Two</title>
+ <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
+ <link rel="icon" href="/favicon.ico" type="image/x-icon">
+
+ <!-- Bootstrap -->
+ <link rel="stylesheet" href="/css/bootstrap.min.css">
+ <link rel="stylesheet" href="/css/flink.css">
+ <link rel="stylesheet" href="/css/syntax.css">
+
+ <!-- Blog RSS feed -->
+ <link href="/blog/feed.xml" rel="alternate" type="application/rss+xml"
title="Apache Flink Blog: RSS feed" />
+
+ <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
+ <!-- We need to load Jquery in the header for custom google analytics
event tracking-->
+ <script src="/js/jquery.min.js"></script>
+
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media
queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script
src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js";></script>
+ <script
src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js";></script>
+ <![endif]-->
+ <!-- Matomo -->
+ <script>
+ var _paq = window._paq = window._paq || [];
+ /* tracker methods like "setCustomDimension" should be called before
"trackPageView" */
+ /* We explicitly disable cookie tracking to avoid privacy issues */
+ _paq.push(['disableCookies']);
+ /* Measure a visit to flink.apache.org and nightlies.apache.org/flink as
the same visit */
+ _paq.push(["setDomains",
["*.flink.apache.org","*.nightlies.apache.org/flink"]]);
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//matomo.privacy.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '1']);
+ var d=document, g=d.createElement('script'),
s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+ <!-- End Matomo Code -->
+ </head>
+ <body>
+
+
+ <!-- Main content. -->
+ <div class="container">
+ <div class="row">
+
+
+ <div id="sidebar" class="col-sm-3">
+
+
+<!-- Top navbar. -->
+ <nav class="navbar navbar-default">
+ <!-- The logo. -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed"
data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <div class="navbar-logo">
+ <a href="/">
+ <img alt="Apache Flink" src="/img/flink-header-logo.svg"
width="147px" height="73px">
+ </a>
+ </div>
+ </div><!-- /.navbar-header -->
+
+ <!-- The navigation links. -->
+ <div class="collapse navbar-collapse"
id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav navbar-main">
+
+ <!-- First menu section explains visitors what Flink is -->
+
+ <!-- What is Stream Processing? -->
+ <!--
+ <li><a href="/streamprocessing1.html">What is Stream
Processing?</a></li>
+ -->
+
+ <!-- What is Flink? -->
+ <li><a href="/flink-architecture.html">What is Apache
Flink?</a></li>
+
+
+
+ <!-- Stateful Functions? -->
+
+ <li><a
href="https://nightlies.apache.org/flink/flink-statefun-docs-stable/";>What is
Stateful Functions?</a></li>
+
+ <!-- Flink ML? -->
+
+ <li><a
href="https://nightlies.apache.org/flink/flink-ml-docs-stable/";>What is Flink
ML?</a></li>
+
+ <!-- Use cases -->
+ <li><a href="/usecases.html">Use Cases</a></li>
+
+ <!-- Powered by -->
+ <li><a href="/poweredby.html">Powered By</a></li>
+
+
+
+ <!-- Second menu section aims to support Flink users -->
+
+ <!-- Downloads -->
+ <li><a href="/downloads.html">Downloads</a></li>
+
+ <!-- Getting Started -->
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown"
href="#">Getting Started<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a
href="https://nightlies.apache.org/flink/flink-docs-release-1.15//docs/try-flink/local_installation/";
target="_blank">With Flink <small><span class="glyphicon
glyphicon-new-window"></span></small></a></li>
+ <li><a
href="https://nightlies.apache.org/flink/flink-statefun-docs-release-3.2/getting-started/project-setup.html";
target="_blank">With Flink Stateful Functions <small><span class="glyphicon
glyphicon-new-window"></span></small></a></li>
+ <li><a
href="https://nightlies.apache.org/flink/flink-ml-docs-release-2.0/try-flink-ml/quick-start.html";
target="_blank">With Flink ML <small><span class="glyphicon
glyphicon-new-window"></span></small></a></li>
+ <li><a
href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-0.1/try-flink-kubernetes-operator/quick-start.html";
target="_blank">With Flink Kubernetes Operator <small><span class="glyphicon
glyphicon-new-window"></span></small></a></li>
+ <li><a
href="https://nightlies.apache.org/flink/flink-table-store-docs-release-0.1/try-table-store/quick-start.html";
target="_blank">With Flink Table Store <small><span class="glyphicon
glyphicon-new-window"></span></small></a></li>
+ <li><a href="/training.html">Training Course</a></li>
+ </ul>
+ </li>
+
+ <!-- Documentation -->
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown"
href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a
href="https://nightlies.apache.org/flink/flink-docs-release-1.15";
target="_blank">Flink 1.15 (Latest stable release) <small><span
class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a
href="https://nightlies.apache.org/flink/flink-docs-master";
target="_blank">Flink Master (Latest Snapshot) <small><span class="glyphicon
glyphicon-new-window"></span></small></a></li>
+ <li><a
href="https://nightlies.apache.org/flink/flink-statefun-docs-release-3.2";
target="_blank">Flink Stateful Functions 3.2 (Latest stable release)
<small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a
href="https://nightlies.apache.org/flink/flink-statefun-docs-master";
target="_blank">Flink Stateful Functions Master (Latest Snapshot) <small><span
class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a
href="https://nightlies.apache.org/flink/flink-ml-docs-release-2.0";
target="_blank">Flink ML 2.0 (Latest stable release) <small><span
class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a
href="https://nightlies.apache.org/flink/flink-ml-docs-master";
target="_blank">Flink ML Master (Latest Snapshot) <small><span class="glyphicon
glyphicon-new-window"></span></small></a></li>
+ <li><a
href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-0.1";
target="_blank">Flink Kubernetes Operator 0.1 (Latest stable release)
<small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a
href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main";
target="_blank">Flink Kubernetes Operator Main (Latest Snapshot) <small><span
class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a
href="https://nightlies.apache.org/flink/flink-table-store-docs-release-0.1";
target="_blank">Flink Table Store 0.1 (Latest stable release) <small><span
class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a
href="https://nightlies.apache.org/flink/flink-table-store-docs-master";
target="_blank">Flink Table Store Master (Latest Snapshot) <small><span
class="glyphicon glyphicon-new-window"></span></small></a></li>
+ </ul>
+ </li>
+
+ <!-- getting help -->
+ <li><a href="/gettinghelp.html">Getting Help</a></li>
+
+ <!-- Blog -->
+ <li><a href="/blog/"><b>Flink Blog</b></a></li>
+
+
+ <!-- Flink-packages -->
+ <li>
+ <a href="https://flink-packages.org";
target="_blank">flink-packages.org <small><span class="glyphicon
glyphicon-new-window"></span></small></a>
+ </li>
+
+
+ <!-- Third menu section aim to support community and contributors
-->
+
+ <!-- Community -->
+ <li><a href="/community.html">Community & Project Info</a></li>
+
+ <!-- Roadmap -->
+ <li><a href="/roadmap.html">Roadmap</a></li>
+
+ <!-- Contribute -->
+ <li><a href="/contributing/how-to-contribute.html">How to
Contribute</a></li>
+
+
+ <!-- GitHub -->
+ <li>
+ <a href="https://github.com/apache/flink"; target="_blank">Flink
on GitHub <small><span class="glyphicon
glyphicon-new-window"></span></small></a>
+ </li>
+
+
+
+ <!-- Language Switcher -->
+ <li>
+
+
+ <a href="/zh/2022/05/23/latency-part2.html">中文版</a>
+
+
+ </li>
+
+ </ul>
+
+ <style>
+ .smalllinks:link {
+ display: inline-block !important; background: none; padding-top:
0px; padding-bottom: 0px; padding-right: 0px; min-width: 75px;
+ }
+ </style>
+
+ <ul class="nav navbar-nav navbar-bottom">
+ <hr />
+
+ <!-- Twitter -->
+ <li><a href="https://twitter.com/apacheflink";
target="_blank">@ApacheFlink <small><span class="glyphicon
glyphicon-new-window"></span></small></a></li>
+
+ <!-- Visualizer -->
+ <li class=" hidden-md hidden-sm"><a href="/visualizer/"
target="_blank">Plan Visualizer <small><span class="glyphicon
glyphicon-new-window"></span></small></a></li>
+
+ <li >
+ <a href="/security.html">Flink Security</a>
+ </li>
+
+ <hr />
+
+ <li><a href="https://apache.org"; target="_blank">Apache Software
Foundation <small><span class="glyphicon
glyphicon-new-window"></span></small></a></li>
+
+ <li>
+
+ <a class="smalllinks" href="https://www.apache.org/licenses/";
target="_blank">License</a> <small><span class="glyphicon
glyphicon-new-window"></span></small>
+
+ <a class="smalllinks" href="https://www.apache.org/security/";
target="_blank">Security</a> <small><span class="glyphicon
glyphicon-new-window"></span></small>
+
+ <a class="smalllinks"
href="https://www.apache.org/foundation/sponsorship.html";
target="_blank">Donate</a> <small><span class="glyphicon
glyphicon-new-window"></span></small>
+
+ <a class="smalllinks"
href="https://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a>
<small><span class="glyphicon glyphicon-new-window"></span></small>
+ </li>
+
+ </ul>
+ </div><!-- /.navbar-collapse -->
+ </nav>
+
+ </div>
+ <div class="col-sm-9">
+ <div class="row-fluid">
+ <div class="col-sm-12">
+ <div class="row">
+ <h1>Getting into Low-Latency Gears with Apache Flink - Part Two</h1>
+ <p><i></i></p>
+
+ <article>
+ <p>23 May 2022 Jun Qin & Nico Kruber </p>
+
+<p>This series of blog posts present a collection of low-latency techniques in
Flink. In <a href="https://flink.apache.org/2022/05/18/latency-part1.html";>part
one</a>, we discussed the types of latency in Flink and the way we measure
end-to-end latency and presented a few techniques that optimize latency
directly. In this post, we will continue with a few more direct latency
optimization techniques. Just like in part one, for each optimization
technique, we will clarify what it is, when [...]
+
+<h1 id="direct-latency-optimization">Direct latency optimization</h1>
+
+<h2 id="spread-work-across-time">Spread work across time</h2>
+
+<p>When you use timers or do windowing in a job, timer or window firing may
create load spikes due to heavy computation or state access. If the allocated
resources cannot cope with these load spikes, timer or window firing will take
a long time to finish. This often results in high latency.</p>
+
+<p>To avoid this situation, you should change your code to spread out the
workload as much as possible such that you do not accumulate too much work to
be done at a single point in time. In the case of windowing, you should
consider using incremental window aggregation with
<code>AggregateFunction</code> or <code>ReduceFunction</code>. In the case of
timers in a <code>ProcessFunction</code>, the operations executed in the
<code>onTimer()</code> method should be optimized such that the ti [...]
+
+<p><strong>You can apply this optimization</strong> if you are using
timer-based processing (e.g., timers, windowing) and an efficient aggregation
can be applied whenever an event arrives instead of waiting for timers to
fire.</p>
+
+<p><strong>Keep in mind</strong> that when you spread work across time, you
should consider not only computation but also state access, especially when
using RocksDB. Spreading one type of work while accumulating the other may
result in higher latencies.</p>
+
+<p><a
href="https://github.com/ververica/lab-flink-latency/blob/main/src/main/java/com/ververica/lablatency/job/WindowingJob.java";>WindowingJob</a>
already does incremental window aggregation with
<code>AggregateFunction</code>. To show the latency improvement of this
technique, we compared <a
href="https://github.com/ververica/lab-flink-latency/blob/main/src/main/java/com/ververica/lablatency/job/WindowingJob.java";>WindowingJob</a>
with a variant that does not do incremental aggregation [...]
+
+<center>
+<img vspace="8" style="width:50%"
src="/img/blog/2022-05-23-latency-part2/spread-work.png" />
+</center>
+
+<h2 id="access-external-systems-efficiently">Access external systems
efficiently</h2>
+
+<h3 id="using-async-io">Using async I/O</h3>
+
+<p>When interacting with external systems (e.g., RDBMS, object stores, web
services) in a Flink job for data enrichment, the latency in getting responses
from external systems often dominates the overall latency of the job. With
Flink’s <a
href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html";>Async
I/O API</a> (e.g., <code>AsyncDataStream.unorderedWait()</code> or
<code>AsyncDataStream.orderedWait()</code>), a single parallel function
instance ca [...]
+
+<center>
+<img vspace="8" style="width:50%"
src="/img/blog/2022-05-23-latency-part2/async-io.png" />
+</center>
+
+<p><strong>You can apply this optimization</strong> if the client of your
external system supports asynchronous requests. If it does not, you can use a
thread pool of multiple clients to handle synchronous requests in parallel. You
can also use a cache to speed up lookups if the data in the external system is
not changing frequently. A cache, however, comes at the cost of working with
outdated data.</p>
+
+<p>In this experiment, we simulated an external system that returns responses
within 1 to 6 ms randomly, and we keep the external system response in a cache
in our job for 1s. The results below show the comparison between two jobs: <a
href="https://github.com/ververica/lab-flink-latency/blob/main/src/main/java/com/ververica/lablatency/job/EnrichingJobSync.java";>EnrichingJobSync</a>
and <a
href="https://github.com/ververica/lab-flink-latency/blob/main/src/main/java/com/ververica/lablatenc
[...]
+
+<center>
+<img vspace="8" style="width:50%"
src="/img/blog/2022-05-23-latency-part2/enriching-with-async-io.png" />
+</center>
+
+<h3 id="using-a-streaming-join">Using a streaming join</h3>
+
+<p>If you are enriching a stream of events with an external database where the
data changes frequently, and the changes can be converted to a data stream,
then you have another option to use <a
href="https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/operators/overview/#datastreamdatastream-rarr-connectedstream";>connected
streams</a> and a <a
href="https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/operators/process_function/#low-level-joins";>C
[...]
+
+<h2 id="tune-checkpointing">Tune checkpointing</h2>
+
+<p>There are two aspects in checkpointing that impact latency: checkpoint
alignment time as well as checkpoint frequency and duration in case of
end-to-end exactly-once with transactional sinks.</p>
+
+<h3 id="reduce-checkpoint-alignment-time">Reduce checkpoint alignment time</h3>
+
+<p>During checkpoint alignment, operators block the event processing from the
channels where checkpoint barriers have been received in order to wait for the
checkpoint barriers from other channels. Longer alignment time will result in
higher latencies.</p>
+
+<p>There are different ways to reduce checkpoint alignment time:</p>
+
+<ul>
+ <li>Improve the throughput. Any improvement in throughput helps processing
the buffers sitting in front of a checkpoint barrier faster.</li>
+ <li>Scale up or scale out. This is the same as the technique of “allocate
enough resources” described in <a
href="https://flink.apache.org/2022/05/18/latency-part1.html";>part one</a>.
Increased processing power helps reducing backpressure and checkpoint alignment
time.</li>
+ <li>Use unaligned checkpointing. In this case, checkpoint barriers will not
wait until the data is processed but skip over and pass on to the next operator
immediately. Skipped-over data, however, has to be checkpointed as well in
order to be consistent. Flink can also be configured to automatically switch
over from aligned to unaligned checkpointing after a certain alignment time has
passed.</li>
+ <li>Buffer less data. You can reduce the buffered data size by tuning the
number of exclusive and floating buffers. With less data buffered in the
network stack, the checkpoint barrier can arrive at operators quicker. However,
reducing buffers has an adverse effect on throughput and is just mentioned here
for completeness. Flink 1.14 improves buffer handling by introducing a feature
called <em>buffer debloating</em>. Buffer debloating can dynamically adjust
buffer size based on the cur [...]
+</ul>
+
+<h3 id="tune-checkpoint-duration-and-frequency">Tune checkpoint duration and
frequency</h3>
+
+<p>If you are working with transactional sinks with exactly-once semantics,
the output events are committed to external systems (e.g., Kafka) <em>only</em>
upon checkpoint completion. In this case, tuning other options may not help if
you do not tune checkpointing. Instead, you need to have fast and more frequent
checkpointing.</p>
+
+<p>To have fast checkpointing, you need to reduce the checkpoint duration. To
achieve that, you can, for example, turn on rocksdb incremental checkpointing,
reduce the state stored in Flink, clean up state that is not needed anymore, do
not put cache into managed state, store only necessary fields in state,
optimize the serialization format, etc. You can also scale up or scale out,
same as the technique of “allocate enough resources” described in <a
href="https://flink.apache.org/2022/05 [...]
+
+<p>To have more frequent checkpointing, you can reduce the checkpoint
interval, the minimum pause between checkpoints, or use concurrent checkpoints.
But keep in mind that concurrent checkpoints introduce more runtime
overhead.</p>
+
+<p>Another option is to not use exactly-once sinks but to switch to
at-least-once sinks. The result of this is that you may have (correct but)
duplicated output events, so this may require the downstream application that
consumes the output events of your jobs to perform deduplication
additionally.</p>
+
+<h2 id="process-events-on-arrival">Process events on arrival</h2>
+<p>In a stream processing pipeline, there often exists a delay between the
time an event is received and the time the event can be processed (e.g., after
having seen all events up to a certain point in event time). The amount of
delay may be significant for those pipelines with very low latency
requirements. For example, a fraud detection job usually requires a sub-second
level of latency. In this case, you could process events with <a
href="https://nightlies.apache.org/flink/flink-docs- [...]
+
+<p><strong>You can apply this optimization</strong> if your job has a
sub-second level latency requirement (e.g., hundreds of milliseconds) and the
reduced watermarking interval still contributes a significant part of the
latency.</p>
+
+<p><strong>Keep in mind</strong> that this may change your job logic
considerably since you have to deal with out-of-order events by yourself.</p>
+
+<h1 id="summary">Summary</h1>
+
+<p>Following part one, this blog post presented a few more latency
optimization techniques with a focus on direct latency optimization. In the
next part, we will focus on techniques that optimize latency by increasing
throughput. Stay tuned!</p>
+
+
+ </article>
+ </div>
+
+ <div class="row">
+ <div id="disqus_thread"></div>
+ <script type="text/javascript">
+ /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE
* * */
+ var disqus_shortname = 'stratosphere-eu'; // required: replace example
with your forum shortname
+
+ /* * * DON'T EDIT BELOW THIS LINE * * */
+ (function() {
+ var dsq = document.createElement('script'); dsq.type =
'text/javascript'; dsq.async = true;
+ dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
+ (document.getElementsByTagName('head')[0] ||
document.getElementsByTagName('body')[0]).appendChild(dsq);
+ })();
+ </script>
+ </div>
+ </div>
+</div>
+ </div>
+ </div>
+
+ <hr />
+
+ <div class="row">
+ <div class="footer text-center col-sm-12">
+ <p>Copyright © 2014-2022 <a href="http://apache.org";>The Apache
Software Foundation</a>. All Rights Reserved.</p>
+ <p>Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache
feather logo are either registered trademarks or trademarks of The Apache
Software Foundation.</p>
+ <p><a
href="https://privacy.apache.org/policies/privacy-policy-public.html";>Privacy
Policy</a> · <a href="/blog/feed.xml">RSS feed</a></p>
+ </div>
+ </div>
+ </div><!-- /.container -->
+
+ <!-- Include all compiled plugins (below), or include individual files as
needed -->
+ <script src="/js/jquery.matchHeight-min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/codetabs.js"></script>
+ <script src="/js/stickysidebar.js"></script>
+ </body>
+</html>
diff --git a/content/blog/feed.xml b/content/blog/feed.xml
index 91ed5cbed..3f1841b3e 100644
--- a/content/blog/feed.xml
+++ b/content/blog/feed.xml
@@ -6,6 +6,94 @@
<link>https://flink.apache.org/blog</link>
<atom:link href="https://flink.apache.org/blog/feed.xml"; rel="self"
type="application/rss+xml" />
+<item>
+<title>Getting into Low-Latency Gears with Apache Flink - Part Two</title>
+<description><p>This series of blog posts present a collection of
low-latency techniques in Flink. In <a
href="https://flink.apache.org/2022/05/18/latency-part1.html">part
one</a>, we discussed the types of latency in Flink and the way we
measure end-to-end latency and presented a few techniques that optimize latency
directly. In this post, we will continue with a few more direct latency
optimization techniques. Just like in part one, for each optimization tec [...]
+
+<h1 id="direct-latency-optimization">Direct latency
optimization</h1>
+
+<h2 id="spread-work-across-time">Spread work across
time</h2>
+
+<p>When you use timers or do windowing in a job, timer or window firing
may create load spikes due to heavy computation or state access. If the
allocated resources cannot cope with these load spikes, timer or window firing
will take a long time to finish. This often results in high latency.</p>
+
+<p>To avoid this situation, you should change your code to spread out
the workload as much as possible such that you do not accumulate too much work
to be done at a single point in time. In the case of windowing, you should
consider using incremental window aggregation with
<code>AggregateFunction</code> or
<code>ReduceFunction</code>. In the case of timers in a
<code>ProcessFunction</code>, the operations executed in the
<code>onTimer()< [...]
+
+<p><strong>You can apply this optimization</strong> if you
are using timer-based processing (e.g., timers, windowing) and an efficient
aggregation can be applied whenever an event arrives instead of waiting for
timers to fire.</p>
+
+<p><strong>Keep in mind</strong> that when you spread work
across time, you should consider not only computation but also state access,
especially when using RocksDB. Spreading one type of work while accumulating
the other may result in higher latencies.</p>
+
+<p><a
href="https://github.com/ververica/lab-flink-latency/blob/main/src/main/java/com/ververica/lablatency/job/WindowingJob.java">WindowingJob</a>
already does incremental window aggregation with
<code>AggregateFunction</code>. To show the latency improvement of
this technique, we compared <a
href="https://github.com/ververica/lab-flink-latency/blob/main/src/main/java/com/ververica/lablatency/job/WindowingJob.java">WindowingJob<
[...]
+
+<center>
+<img vspace="8" style="width:50%"
src="/img/blog/2022-05-23-latency-part2/spread-work.png" />
+</center>
+
+<h2 id="access-external-systems-efficiently">Access external
systems efficiently</h2>
+
+<h3 id="using-async-io">Using async I/O</h3>
+
+<p>When interacting with external systems (e.g., RDBMS, object stores,
web services) in a Flink job for data enrichment, the latency in getting
responses from external systems often dominates the overall latency of the job.
With Flink’s <a
href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html">Async
I/O API</a> (e.g.,
<code>AsyncDataStream.unorderedWait()</code> or
<code>AsyncDataStream.orderedWait()&l [...]
+
+<center>
+<img vspace="8" style="width:50%"
src="/img/blog/2022-05-23-latency-part2/async-io.png" />
+</center>
+
+<p><strong>You can apply this optimization</strong> if the
client of your external system supports asynchronous requests. If it does not,
you can use a thread pool of multiple clients to handle synchronous requests in
parallel. You can also use a cache to speed up lookups if the data in the
external system is not changing frequently. A cache, however, comes at the cost
of working with outdated data.</p>
+
+<p>In this experiment, we simulated an external system that returns
responses within 1 to 6 ms randomly, and we keep the external system response
in a cache in our job for 1s. The results below show the comparison between two
jobs: <a
href="https://github.com/ververica/lab-flink-latency/blob/main/src/main/java/com/ververica/lablatency/job/EnrichingJobSync.java">EnrichingJobSync</a>
and <a
href="https://github.com/ververica/lab-flink-latency/blob/main/s [...]
+
+<center>
+<img vspace="8" style="width:50%"
src="/img/blog/2022-05-23-latency-part2/enriching-with-async-io.png"
/>
+</center>
+
+<h3 id="using-a-streaming-join">Using a streaming
join</h3>
+
+<p>If you are enriching a stream of events with an external database
where the data changes frequently, and the changes can be converted to a data
stream, then you have another option to use <a
href="https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/operators/overview/#datastreamdatastream-rarr-connectedstream">connected
streams</a> and a <a
href="https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/operators/
[...]
+
+<h2 id="tune-checkpointing">Tune checkpointing</h2>
+
+<p>There are two aspects in checkpointing that impact latency:
checkpoint alignment time as well as checkpoint frequency and duration in case
of end-to-end exactly-once with transactional sinks.</p>
+
+<h3 id="reduce-checkpoint-alignment-time">Reduce checkpoint
alignment time</h3>
+
+<p>During checkpoint alignment, operators block the event processing
from the channels where checkpoint barriers have been received in order to wait
for the checkpoint barriers from other channels. Longer alignment time will
result in higher latencies.</p>
+
+<p>There are different ways to reduce checkpoint alignment
time:</p>
+
+<ul>
+ <li>Improve the throughput. Any improvement in throughput helps
processing the buffers sitting in front of a checkpoint barrier
faster.</li>
+ <li>Scale up or scale out. This is the same as the technique of
“allocate enough resources” described in <a
href="https://flink.apache.org/2022/05/18/latency-part1.html">part
one</a>. Increased processing power helps reducing backpressure and
checkpoint alignment time.</li>
+ <li>Use unaligned checkpointing. In this case, checkpoint barriers
will not wait until the data is processed but skip over and pass on to the next
operator immediately. Skipped-over data, however, has to be checkpointed as
well in order to be consistent. Flink can also be configured to automatically
switch over from aligned to unaligned checkpointing after a certain alignment
time has passed.</li>
+ <li>Buffer less data. You can reduce the buffered data size by tuning
the number of exclusive and floating buffers. With less data buffered in the
network stack, the checkpoint barrier can arrive at operators quicker. However,
reducing buffers has an adverse effect on throughput and is just mentioned here
for completeness. Flink 1.14 improves buffer handling by introducing a feature
called <em>buffer debloating</em>. Buffer debloating can
dynamically adjust buffer siz [...]
+</ul>
+
+<h3 id="tune-checkpoint-duration-and-frequency">Tune
checkpoint duration and frequency</h3>
+
+<p>If you are working with transactional sinks with exactly-once
semantics, the output events are committed to external systems (e.g., Kafka)
<em>only</em> upon checkpoint completion. In this case, tuning
other options may not help if you do not tune checkpointing. Instead, you need
to have fast and more frequent checkpointing.</p>
+
+<p>To have fast checkpointing, you need to reduce the checkpoint
duration. To achieve that, you can, for example, turn on rocksdb incremental
checkpointing, reduce the state stored in Flink, clean up state that is not
needed anymore, do not put cache into managed state, store only necessary
fields in state, optimize the serialization format, etc. You can also scale up
or scale out, same as the technique of “allocate enough resources” described in
<a href="https://flink.apac [...]
+
+<p>To have more frequent checkpointing, you can reduce the checkpoint
interval, the minimum pause between checkpoints, or use concurrent checkpoints.
But keep in mind that concurrent checkpoints introduce more runtime
overhead.</p>
+
+<p>Another option is to not use exactly-once sinks but to switch to
at-least-once sinks. The result of this is that you may have (correct but)
duplicated output events, so this may require the downstream application that
consumes the output events of your jobs to perform deduplication
additionally.</p>
+
+<h2 id="process-events-on-arrival">Process events on
arrival</h2>
+<p>In a stream processing pipeline, there often exists a delay between
the time an event is received and the time the event can be processed (e.g.,
after having seen all events up to a certain point in event time). The amount
of delay may be significant for those pipelines with very low latency
requirements. For example, a fraud detection job usually requires a sub-second
level of latency. In this case, you could process events with <a
href="https://nightlies.apache.org/fli [...]
+
+<p><strong>You can apply this optimization</strong> if your
job has a sub-second level latency requirement (e.g., hundreds of milliseconds)
and the reduced watermarking interval still contributes a significant part of
the latency.</p>
+
+<p><strong>Keep in mind</strong> that this may change your
job logic considerably since you have to deal with out-of-order events by
yourself.</p>
+
+<h1 id="summary">Summary</h1>
+
+<p>Following part one, this blog post presented a few more latency
optimization techniques with a focus on direct latency optimization. In the
next part, we will focus on techniques that optimize latency by increasing
throughput. Stay tuned!</p>
+
+</description>
+<pubDate>Mon, 23 May 2022 02:00:00 +0200</pubDate>
+<link>https://flink.apache.org/2022/05/23/latency-part2.html</link>
+<guid isPermaLink="true">/2022/05/23/latency-part2.html</guid>
+</item>
+
<item>
<title>Getting into Low-Latency Gears with Apache Flink - Part One</title>
<description><p>Apache Flink is a stream processing framework well known
for its low latency processing capabilities. It is generic and suitable for a
wide range of use cases. As a Flink application developer or a cluster
administrator, you need to find the right gear that is best for your
application. In other words, you don’t want to be driving a luxury sports car
while only using the first gear.</p>
@@ -19982,365 +20070,5 @@ zhangxin516, zhangxinxing, zhaofaxian, zhijiang,
zjuwangg, 林小铂,
<guid isPermaLink="true">/news/2019/08/22/release-1.9.0.html</guid>
</item>
-<item>
-<title>Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure
Thing</title>
-<description><style type="text/css">
-.tg {border-collapse:collapse;border-spacing:0;}
-.tg td{padding:10px
10px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}
-.tg th{padding:10px
10px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;background-color:#eff0f1;}
-.tg .tg-wide{padding:10px 30px;}
-.tg .tg-top{vertical-align:top}
-.tg .tg-topcenter{text-align:center;vertical-align:top}
-.tg .tg-center{text-align:center;vertical-align:center}
-</style>
-
-<p>In a <a
href="/2019/06/05/flink-network-stack.html">previous blog
post</a>, we presented how Flink’s network stack works from the
high-level abstractions to the low-level details. This second blog post in the
series of network stack posts extends on this knowledge and discusses
monitoring network-related metrics to identify effects such as backpressure or
bottlenecks in throughput and latency. Although this post briefly covers what
to do with backpressure, [...]
-
-<div class="page-toc">
-<ul id="markdown-toc">
- <li><a href="#monitoring"
id="markdown-toc-monitoring">Monitoring</a> <ul>
- <li><a href="#backpressure-monitor"
id="markdown-toc-backpressure-monitor">Backpressure
Monitor</a></li>
- </ul>
- </li>
- <li><a href="#network-metrics"
id="markdown-toc-network-metrics">Network Metrics</a>
<ul>
- <li><a href="#backpressure"
id="markdown-toc-backpressure">Backpressure</a></li>
- <li><a href="#resource-usage--throughput"
id="markdown-toc-resource-usage--throughput">Resource Usage /
Throughput</a></li>
- <li><a href="#latency-tracking"
id="markdown-toc-latency-tracking">Latency
Tracking</a></li>
- </ul>
- </li>
- <li><a href="#conclusion"
id="markdown-toc-conclusion">Conclusion</a></li>
-</ul>
-
-</div>
-
-<h2 id="monitoring">Monitoring</h2>
-
-<p>Probably the most important part of network monitoring is <a
href="https://nightlies.apache.org/flink/flink-docs-release-1.8/monitoring/back_pressure.html">monitoring
backpressure</a>, a situation where a system is receiving data at a
higher rate than it can process¹. Such behaviour will result in the sender
being backpressured and may be caused by two things:</p>
-
-<ul>
- <li>
- <p>The receiver is slow.<br />
-This can happen because the receiver is backpressured itself, is unable to
keep processing at the same rate as the sender, or is temporarily blocked by
garbage collection, lack of system resources, or I/O.</p>
- </li>
- <li>
- <p>The network channel is slow.<br />
- Even though in such case the receiver is not (directly) involved, we call
the sender backpressured due to a potential oversubscription on network
bandwidth shared by all subtasks running on the same machine. Beware that, in
addition to Flink’s network stack, there may be more network users, such as
sources and sinks, distributed file systems (checkpointing, network-attached
storage), logging, and metrics. A previous <a
href="https://www.ververica.com/blog/how-to-size-your-apach [...]
- </li>
-</ul>
-
-<p><sup>1</sup> In case you are unfamiliar with backpressure
and how it interacts with Flink, we recommend reading through <a
href="https://www.ververica.com/blog/how-flink-handles-backpressure">this
blog post on backpressure</a> from 2015.</p>
-
-<p><br />
-If backpressure occurs, it will bubble upstream and eventually reach your
sources and slow them down. This is not a bad thing per-se and merely states
that you lack resources for the current load. However, you may want to improve
your job so that it can cope with higher loads without using more resources. In
order to do so, you need to find (1) where (at which task/operator) the
bottleneck is and (2) what is causing it. Flink offers two mechanisms for
identifying where the bottleneck is: [...]
-
-<ul>
- <li>directly via Flink’s web UI and its backpressure monitor,
or</li>
- <li>indirectly through some of the network metrics.</li>
-</ul>
-
-<p>Flink’s web UI is likely the first entry point for a quick
troubleshooting but has some disadvantages that we will explain below. On the
other hand, Flink’s network metrics are better suited for continuous monitoring
and reasoning about the exact nature of the bottleneck causing backpressure. We
will cover both in the sections below. In both cases, you need to identify the
origin of backpressure from the sources to the sinks. Your starting point for
the current and future invest [...]
-
-<h3 id="backpressure-monitor">Backpressure Monitor</h3>
-
-<p>The <a
href="https://nightlies.apache.org/flink/flink-docs-release-1.8/monitoring/back_pressure.html">backpressure
monitor</a> is only exposed via Flink’s web UI². Since it’s an active
component that is only triggered on request, it is currently not available via
metrics. The backpressure monitor samples the running tasks’ threads on all
TaskManagers via <code>Thread.getStackTrace()</code> and computes
the number of samples where tasks were bloc [...]
-
-<ul>
- <li><span style="color:green">OK</span> for
<code>ratio ≤ 0.10</code>,</li>
- <li><span style="color:orange">LOW</span> for
<code>0.10 &lt; Ratio ≤ 0.5</code>, and</li>
- <li><span style="color:red">HIGH</span> for
<code>0.5 &lt; Ratio ≤ 1</code>.</li>
-</ul>
-
-<p>Although you can tune things like the refresh-interval, the number of
samples, or the delay between samples, normally, you would not need to touch
these since the defaults already give good-enough results.</p>
-
-<center>
-<img
src="/img/blog/2019-07-23-network-stack-2/back_pressure_sampling_high.png"
width="600px" alt="Backpressure sampling:high" />
-</center>
-
-<p><sup>2</sup> You may also access the backpressure monitor
via the REST API:
<code>/jobs/:jobid/vertices/:vertexid/backpressure</code></p>
-
-<p><br />
-The backpressure monitor can help you find where (at which task/operator)
backpressure originates from. However, it does not support you in further
reasoning about the causes of it. Additionally, for larger jobs or higher
parallelism, the backpressure monitor becomes too crowded to use and may also
take some time to gather all information from all TaskManagers. Please also
note that sampling may affect your running job’s performance.</p>
-
-<h2 id="network-metrics">Network Metrics</h2>
-
-<p><a
href="https://nightlies.apache.org/flink/flink-docs-release-1.8/monitoring/metrics.html#network">Network</a>
and <a
href="https://nightlies.apache.org/flink/flink-docs-release-1.8/monitoring/metrics.html#io">task
I/O</a> metrics are more lightweight than the backpressure monitor and
are continuously published for each running job. We can leverage those and get
even more insights, not only for backpressure monitoring. The most releva [...]
-
-<ul>
- <li>
- <p><strong><span style="color:orange">up to
Flink 1.8:</span></strong> <code>outPoolUsage</code>,
<code>inPoolUsage</code><br />
-An estimate on the ratio of buffers used vs. buffers available in the
respective local buffer pools.
-While interpreting <code>inPoolUsage</code> in Flink 1.5 - 1.8
with credit-based flow control, please note that this only relates to floating
buffers (exclusive buffers are not part of the pool).</p>
- </li>
- <li>
- <p><strong><span style="color:green">Flink 1.9
and above:</span></strong> <code>outPoolUsage</code>,
<code>inPoolUsage</code>,
<code>floatingBuffersUsage</code>,
<code>exclusiveBuffersUsage</code><br />
-An estimate on the ratio of buffers used vs. buffers available in the
respective local buffer pools.
-Starting with Flink 1.9, <code>inPoolUsage</code> is the sum of
<code>floatingBuffersUsage</code> and
<code>exclusiveBuffersUsage</code>.</p>
- </li>
- <li>
- <p><code>numRecordsOut</code>,
<code>numRecordsIn</code><br />
-Each metric comes with two scopes: one scoped to the operator and one scoped
to the subtask. For network monitoring, the subtask-scoped metric is relevant
and shows the total number of records it has sent/received. You may need to
further look into these figures to extract the number of records within a
certain time span or use the equivalent <code>…PerSecond</code>
metrics.</p>
- </li>
- <li>
- <p><code>numBytesOut</code>,
<code>numBytesInLocal</code>,
<code>numBytesInRemote</code><br />
-The total number of bytes this subtask has emitted or read from a local/remote
source. These are also available as meters via
<code>…PerSecond</code> metrics.</p>
- </li>
- <li>
- <p><code>numBuffersOut</code>,
<code>numBuffersInLocal</code>,
<code>numBuffersInRemote</code><br />
-Similar to <code>numBytes…</code> but counting the number of
network buffers.</p>
- </li>
-</ul>
-
-<div class="alert alert-warning">
- <p><span class="label label-warning" style="display:
inline-block"><span class="glyphicon
glyphicon-warning-sign" aria-hidden="true"></span>
Warning</span>
-For the sake of completeness and since they have been used in the past, we
will briefly look at the <code>outputQueueLength</code> and
<code>inputQueueLength</code> metrics. These are somewhat similar
to the <code>[out,in]PoolUsage</code> metrics but show the number
of buffers sitting in a sender subtask’s output queues and in a receiver
subtask’s input queues, respectively. Reasoning about absolute numbers of
buffers, however, is difficult and there i [...]
-
- <p>Overall, <strong>we discourage the use of</strong>
<code>outputQueueLength</code> <strong>and</strong>
<code>inputQueueLength</code> because their interpretation highly
depends on the current parallelism of the operator and the configured numbers
of exclusive and floating buffers. Instead, we recommend using the various
<code>*PoolUsage</code> metrics which even reveal more detailed
insight.</p>
-</div>
-
-<div class="alert alert-info">
- <p><span class="label label-info" style="display:
inline-block"><span class="glyphicon glyphicon-info-sign"
aria-hidden="true"></span> Note</span>
- If you reason about buffer usage, please keep the following in mind:</p>
-
- <ul>
- <li>Any outgoing channel which has been used at least once will
always occupy one buffer (since Flink 1.5).
- <ul>
- <li><strong><span style="color:orange">up
to Flink 1.8:</span></strong> This buffer (even if empty!) was
always counted as a backlog of 1 and thus receivers tried to reserve a floating
buffer for it.</li>
- <li><strong><span
style="color:green">Flink 1.9 and
above:</span></strong> A buffer is only counted in the backlog if
it is ready for consumption, i.e. it is full or was flushed (see
FLINK-11082)</li>
- </ul>
- </li>
- <li>The receiver will only release a received buffer after
deserialising the last record in it.</li>
- </ul>
-</div>
-
-<p>The following sections make use of and combine these metrics to
reason about backpressure and resource usage / efficiency with respect to
throughput. A separate section will detail latency related metrics.</p>
-
-<h3 id="backpressure">Backpressure</h3>
-
-<p>Backpressure may be indicated by two different sets of metrics:
(local) buffer pool usages as well as input/output queue lengths. They provide
a different level of granularity but, unfortunately, none of these are
exhaustive and there is room for interpretation. Because of the inherent
problems with interpreting these queue lengths we will focus on the usage of
input and output pools below which also provides more detail.</p>
-
-<ul>
- <li>
- <p><strong>If a subtask’s</strong>
<code>outPoolUsage</code> <strong>is 100%</strong>, it
is backpressured. Whether the subtask is already blocking or still writing
records into network buffers depends on how full the buffers are, that the
<code>RecordWriters</code> are currently writing into.<br />
-<span class="glyphicon glyphicon-warning-sign"
aria-hidden="true" style="color:orange;"></span>
This is different to what the backpressure monitor is showing!</p>
- </li>
- <li>
- <p>An <code>inPoolUsage</code> of 100% means that all
floating buffers are assigned to channels and eventually backpressure will be
exercised upstream. These floating buffers are in either of the following
conditions: they are reserved for future use on a channel due to an exclusive
buffer being utilised (remote input channels always try to maintain
<code>#exclusive buffers</code> credits), they are reserved for a
sender’s backlog and wait for data, they [...]
- </li>
- <li>
- <p><strong><span style="color:orange">up to
Flink 1.8:</span></strong> Due to <a
href="https://issues.apache.org/jira/browse/FLINK-11082">FLINK-11082</a>,
an <code>inPoolUsage</code> of 100% is quite common even in normal
situations.</p>
- </li>
- <li>
- <p><strong><span style="color:green">Flink 1.9
and above:</span></strong> If <code>inPoolUsage</code>
is constantly around 100%, this is a strong indicator for exercising
backpressure upstream.</p>
- </li>
-</ul>
-
-<p>The following table summarises all combinations and their
interpretation. Bear in mind, though, that backpressure may be minor or
temporary (no need to look into it), on particular channels only, or caused by
other JVM processes on a particular TaskManager, such as GC, synchronisation,
I/O, resource shortage, instead of a specific subtask.</p>
-
-<center>
-<table class="tg">
- <tr>
- <th></th>
- <th
class="tg-center"><code>outPoolUsage</code>
low</th>
- <th
class="tg-center"><code>outPoolUsage</code>
high</th>
- </tr>
- <tr>
- <th class="tg-top"><code>inPoolUsage</code>
low</th>
- <td class="tg-topcenter">
- <span class="glyphicon glyphicon-ok-sign"
aria-hidden="true"
style="color:green;font-size:1.5em;"></span></td>
- <td class="tg-topcenter">
- <span class="glyphicon glyphicon-warning-sign"
aria-hidden="true"
style="color:orange;font-size:1.5em;"></span><br />
- (backpressured, temporary situation: upstream is not backpressured yet
or not anymore)</td>
- </tr>
- <tr>
- <th class="tg-top" rowspan="2">
- <code>inPoolUsage</code> high<br />
- (<strong><span style="color:green">Flink
1.9+</span></strong>)</th>
- <td class="tg-topcenter">
- if all upstream tasks’<code>outPoolUsage</code> are low:
<span class="glyphicon glyphicon-warning-sign"
aria-hidden="true"
style="color:orange;font-size:1.5em;"></span><br />
- (may eventually cause backpressure)</td>
- <td class="tg-topcenter" rowspan="2">
- <span class="glyphicon glyphicon-remove-sign"
aria-hidden="true"
style="color:red;font-size:1.5em;"></span><br />
- (backpressured by downstream task(s) or network, probably forwarding
backpressure upstream)</td>
- </tr>
- <tr>
- <td class="tg-topcenter">if any upstream
task’s<code>outPoolUsage</code> is high: <span
class="glyphicon glyphicon-remove-sign" aria-hidden="true"
style="color:red;font-size:1.5em;"></span><br />
- (may exercise backpressure upstream and may be the source of
backpressure)</td>
- </tr>
-</table>
-</center>
-
-<p><br />
-We may even reason more about the cause of backpressure by looking at the
network metrics of the subtasks of two consecutive tasks:</p>
-
-<ul>
- <li>If all subtasks of the receiver task have low
<code>inPoolUsage</code> values and any upstream subtask’s
<code>outPoolUsage</code> is high, then there may be a network
bottleneck causing backpressure.
-Since network is a shared resource among all subtasks of a TaskManager, this
may not directly originate from this subtask, but rather from various
concurrent operations, e.g. checkpoints, other streams, external connections,
or other TaskManagers/processes on the same machine.</li>
-</ul>
-
-<p>Backpressure can also be caused by all parallel instances of a task
or by a single task instance. The first usually happens because the task is
performing some time consuming operation that applies to all input partitions.
The latter is usually the result of some kind of skew, either data skew or
resource availability/allocation skew. In either case, you can find some hints
on how to handle such situations in the <a
href="#span-classlabel-label-info-styledisplay-inline-b [...]
-
-<div class="alert alert-info">
- <h3 class="no_toc"
id="span-classglyphicon-glyphicon-info-sign-aria-hiddentruespan-flink-19-and-above"><span
class="glyphicon glyphicon-info-sign"
aria-hidden="true"></span> Flink 1.9 and above</h3>
-
- <ul>
- <li>If <code>floatingBuffersUsage</code> is not 100%, it
is unlikely that there is backpressure. If it is 100% and any upstream task is
backpressured, it suggests that this input is exercising backpressure on either
a single, some or all input channels. To differentiate between those three
situations you can use <code>exclusiveBuffersUsage</code>:
- <ul>
- <li>Assuming that <code>floatingBuffersUsage</code>
is around 100%, the higher the <code>exclusiveBuffersUsage</code>
the more input channels are backpressured. In an extreme case of
<code>exclusiveBuffersUsage</code> being close to 100%, it means
that all channels are backpressured.</li>
- </ul>
- </li>
- </ul>
-
- <p><br />
-The relation between <code>exclusiveBuffersUsage</code>,
<code>floatingBuffersUsage</code>, and the upstream tasks’
<code>outPoolUsage</code> is summarised in the following table and
extends on the table above with <code>inPoolUsage = floatingBuffersUsage
+ exclusiveBuffersUsage</code>:</p>
-
- <center>
-<table class="tg">
- <tr>
- <th></th>
- <th><code>exclusiveBuffersUsage</code> low</th>
- <th><code>exclusiveBuffersUsage</code> high</th>
- </tr>
- <tr>
- <th class="tg-top" style="min-width:33%;">
- <code>floatingBuffersUsage</code> low +<br />
- <em>all</em> upstream <code>outPoolUsage</code>
low</th>
- <td class="tg-center"><span class="glyphicon
glyphicon-ok-sign" aria-hidden="true"
style="color:green;font-size:1.5em;"></span></td>
- <td class="tg-center">-<sup>3</sup></td>
- </tr>
- <tr>
- <th class="tg-top" style="min-width:33%;">
- <code>floatingBuffersUsage</code> low +<br />
- <em>any</em> upstream <code>outPoolUsage</code>
high</th>
- <td class="tg-center">
- <span class="glyphicon glyphicon-remove-sign"
aria-hidden="true"
style="color:red;font-size:1.5em;"></span><br />
- (potential network bottleneck)</td>
- <td class="tg-center">-<sup>3</sup></td>
- </tr>
- <tr>
- <th class="tg-top" style="min-width:33%;">
- <code>floatingBuffersUsage</code> high +<br />
- <em>all</em> upstream <code>outPoolUsage</code>
low</th>
- <td class="tg-center">
- <span class="glyphicon glyphicon-warning-sign"
aria-hidden="true"
style="color:orange;font-size:1.5em;"></span><br />
- (backpressure eventually appears on only some of the input
channels)</td>
- <td class="tg-center">
- <span class="glyphicon glyphicon-warning-sign"
aria-hidden="true"
style="color:orange;font-size:1.5em;"></span><br />
- (backpressure eventually appears on most or all of the input
channels)</td>
- </tr>
- <tr>
- <th class="tg-top" style="min-width:33%;">
- <code>floatingBuffersUsage</code> high +<br />
- any upstream <code>outPoolUsage</code> high</th>
- <td class="tg-center">
- <span class="glyphicon glyphicon-remove-sign"
aria-hidden="true"
style="color:red;font-size:1.5em;"></span><br />
- (backpressure on only some of the input channels)</td>
- <td class="tg-center">
- <span class="glyphicon glyphicon-remove-sign"
aria-hidden="true"
style="color:red;font-size:1.5em;"></span><br />
- (backpressure on most or all of the input channels)</td>
- </tr>
-</table>
-</center>
-
- <p><sup>3</sup> this should not happen</p>
-
-</div>
-
-<h3 id="resource-usage--throughput">Resource Usage /
Throughput</h3>
-
-<p>Besides the obvious use of each individual metric mentioned above,
there are also a few combinations providing useful insight into what is
happening in the network stack:</p>
-
-<ul>
- <li>
- <p>Low throughput with frequent
<code>outPoolUsage</code> values around 100% but low
<code>inPoolUsage</code> on all receivers is an indicator that the
round-trip-time of our credit-notification (depends on your network’s latency)
is too high for the default number of exclusive buffers to make use of your
bandwidth. Consider increasing the <a
href="https://nightlies.apache.org/flink/flink-docs-release-1.8/ops/config.html#taskmanager-network-memor
[...]
- </li>
- <li>
- <p>Combining <code>numRecordsOut</code> and
<code>numBytesOut</code> helps identifying average serialised
record sizes which supports you in capacity planning for peak
scenarios.</p>
- </li>
- <li>
- <p>If you want to reason about buffer fill rates and the influence
of the output flusher, you may combine
<code>numBytesInRemote</code> with
<code>numBuffersInRemote</code>. When tuning for throughput (and
not latency!), low buffer fill rates may indicate reduced network efficiency.
In such cases, consider increasing the buffer timeout.
-Please note that, as of Flink 1.8 and 1.9,
<code>numBuffersOut</code> only increases for buffers getting full
or for an event cutting off a buffer (e.g. a checkpoint barrier) and may lag
behind. Please also note that reasoning about buffer fill rates on local
channels is unnecessary since buffering is an optimisation technique for remote
channels with limited effect on local channels.</p>
- </li>
- <li>
- <p>You may also separate local from remote traffic using
numBytesInLocal and numBytesInRemote but in most cases this is
unnecessary.</p>
- </li>
-</ul>
-
-<div class="alert alert-info">
- <h3 class="no_toc"
id="span-classglyphicon-glyphicon-info-sign-aria-hiddentruespan-what-to-do-with-backpressure"><span
class="glyphicon glyphicon-info-sign"
aria-hidden="true"></span> What to do with
Backpressure?</h3>
-
- <p>Assuming that you identified where the source of backpressure — a
bottleneck — is located, the next step is to analyse why this is happening.
Below, we list some potential causes of backpressure from the more basic to the
more complex ones. We recommend to check the basic causes first, before diving
deeper on the more complex ones and potentially drawing false
conclusions.</p>
-
- <p>Please also recall that backpressure might be temporary and the
result of a load spike, checkpointing, or a job restart with a data backlog
waiting to be processed. In that case, you can often just ignore it.
Alternatively, keep in mind that the process of analysing and solving the issue
can be affected by the intermittent nature of your bottleneck. Having said
that, here are a couple of things to check.</p>
-
- <h4 id="system-resources">System Resources</h4>
-
- <p>Firstly, you should check the incriminated machines’ basic resource
usage like CPU, network, or disk I/O. If some resource is fully or heavily
utilised you can do one of the following:</p>
-
- <ol>
- <li>Try to optimise your code. Code profilers are helpful in this
case.</li>
- <li>Tune Flink for that specific resource.</li>
- <li>Scale out by increasing the parallelism and/or increasing the
number of machines in the cluster.</li>
- </ol>
-
- <h4 id="garbage-collection">Garbage Collection</h4>
-
- <p>Oftentimes, performance issues arise from long GC pauses. You can
verify whether you are in such a situation by either printing debug GC logs
(via -<code>XX:+PrintGCDetails</code>) or by using some memory/GC
profilers. Since dealing with GC issues is highly application-dependent and
independent of Flink, we will not go into details here (<a
href="https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/index.html">Oracle’s
Garbage Collecti [...]
-
- <h4 id="cputhread-bottleneck">CPU/Thread
Bottleneck</h4>
-
- <p>Sometimes a CPU bottleneck might not be visible at first glance if
one or a couple of threads are causing the CPU bottleneck while the CPU usage
of the overall machine remains relatively low. For instance, a single
CPU-bottlenecked thread on a 48-core machine would result in only 2% CPU use.
Consider using code profilers for this as they can identify hot threads by
showing each threads’ CPU usage, for example.</p>
-
- <h4 id="thread-contention">Thread Contention</h4>
-
- <p>Similarly to the CPU/thread bottleneck issue above, a subtask may
be bottlenecked due to high thread contention on shared resources. Again, CPU
profilers are your best friend here! Consider looking for synchronisation
overhead / lock contention in user code — although adding synchronisation in
user code should be avoided and may even be dangerous! Also consider
investigating shared system resources. The default JVM’s SSL implementation,
for example, can become contented around [...]
-
- <h4 id="load-imbalance">Load Imbalance</h4>
-
- <p>If your bottleneck is caused by data skew, you can try to remove it
or mitigate its impact by changing the data partitioning to separate heavy keys
or by implementing local/pre-aggregation.</p>
-
- <p><br />
-This list is far from exhaustive. Generally, in order to reduce a bottleneck
and thus backpressure, first analyse where it is happening and then find out
why. The best place to start reasoning about the “why” is by checking what
resources are fully utilised.</p>
-</div>
-
-<h3 id="latency-tracking">Latency Tracking</h3>
-
-<p>Tracking latencies at the various locations they may occur is a topic
of its own. In this section, we will focus on the time records wait inside
Flink’s network stack — including the system’s network connections. In low
throughput scenarios, these latencies are influenced directly by the output
flusher via the buffer timeout parameter or indirectly by any application code
latencies. When processing a record takes longer than expected or when
(multiple) timers fire at the same ti [...]
-
-<p>Flink offers some support for <a
href="https://nightlies.apache.org/flink/flink-docs-release-1.8/monitoring/metrics.html#latency-tracking">tracking
the latency</a> of records passing through the system (outside of user
code). However, this is disabled by default (see below why!) and must be
enabled by setting a latency tracking interval either in Flink’s <a
href="https://nightlies.apache.org/flink/flink-docs-release-1.8/ops/config.html#metrics-laten
[...]
-
-<ul>
- <li><code>single</code>: one histogram for each operator
subtask</li>
- <li><code>operator</code> (default): one histogram for
each combination of source task and operator subtask</li>
- <li><code>subtask</code>: one histogram for each
combination of source subtask and operator subtask (quadratic in the
parallelism!)</li>
-</ul>
-
-<p>These metrics are collected through special “latency markers”: each
source subtask will periodically emit a special record containing the timestamp
of its creation. The latency markers then flow alongside normal records while
not overtaking them on the wire or inside a buffer queue. However, <em>a
latency marker does not enter application logic</em> and is overtaking
records there. Latency markers therefore only measure the waiting time between
the user code and not [...]
-
-<p>Since <code>LatencyMarkers</code> sit in network buffers
just like normal records, they will also wait for the buffer to be full or
flushed due to buffer timeouts. When a channel is on high load, there is no
added latency by the network buffering data. However, as soon as one channel is
under low load, records and latency markers will experience an expected average
delay of at most <code>buffer_timeout / 2</code>. This delay will
add to each network conne [...]
-
-<p>By looking at the exposed latency tracking metrics for each subtask,
for example at the 95th percentile, you should nevertheless be able to identify
subtasks which are adding substantially to the overall source-to-sink latency
and continue with optimising there.</p>
-
-<div class="alert alert-info">
- <p><span class="label label-info" style="display:
inline-block"><span class="glyphicon glyphicon-info-sign"
aria-hidden="true"></span> Note</span>
-Flink’s latency markers assume that the clocks on all machines in the cluster
are in sync. We recommend setting up an automated clock synchronisation service
(like NTP) to avoid false latency results.</p>
-</div>
-
-<div class="alert alert-warning">
- <p><span class="label label-warning" style="display:
inline-block"><span class="glyphicon
glyphicon-warning-sign" aria-hidden="true"></span>
Warning</span>
-Enabling latency metrics can significantly impact the performance of the
cluster (in particular for <code>subtask</code> granularity) due to
the sheer amount of metrics being added as well as the use of histograms which
are quite expensive to maintain. It is highly recommended to only use them for
debugging purposes.</p>
-</div>
-
-<h2 id="conclusion">Conclusion</h2>
-
-<p>In the previous sections we discussed how to monitor Flink’s network
stack which primarily involves identifying backpressure: where it occurs, where
it originates from, and (potentially) why it occurs. This can be executed in
two ways: for simple cases and debugging sessions by using the backpressure
monitor; for continuous monitoring, more in-depth analysis, and less runtime
overhead by using Flink’s task and network stack metrics. Backpressure can be
caused by the network laye [...]
-
-<p>Stay tuned for the third blog post in the series of network stack
posts that will focus on tuning techniques and anti-patterns to avoid.</p>
-
-</description>
-<pubDate>Tue, 23 Jul 2019 17:30:00 +0200</pubDate>
-<link>https://flink.apache.org/2019/07/23/flink-network-stack-2.html</link>
-<guid isPermaLink="true">/2019/07/23/flink-network-stack-2.html</guid>
-</item>
-
</channel>
</rss>
diff --git a/content/blog/index.html b/content/blog/index.html
index 58bb36c63..a53f7a637 100644
--- a/content/blog/index.html
+++ b/content/blog/index.html
@@ -232,6 +232,19 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/2022/05/23/latency-part2.html">Getting
into Low-Latency Gears with Apache Flink - Part Two</a></h2>
+
+ <p>23 May 2022
+ Jun Qin & Nico Kruber </p>
+
+ <p>This multi-part series of blog post presents a collection of
low-latency techniques in Flink. Following with part one, Part two continues
with a few more techniques that optimize latency directly.</p>
+
+ <p><a href="/2022/05/23/latency-part2.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/2022/05/18/latency-part1.html">Getting
into Low-Latency Gears with Apache Flink - Part One</a></h2>
@@ -360,19 +373,6 @@ exciting changes.</p>
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2022/02/18/release-1.13.6.html">Apache Flink 1.13.6 Release
Announcement</a></h2>
-
- <p>18 Feb 2022
- Konstantin Knauf (<a
href="https://twitter.com/snntrable";>@snntrable</a>)</p>
-
- <p>The Apache Flink Community is please to announce another bug fix
release for Flink 1.13.</p>
-
- <p><a href="/news/2022/02/18/release-1.13.6.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -405,6 +405,16 @@ exciting changes.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/blog/page10/index.html b/content/blog/page10/index.html
index e08af5517..4cdfeed15 100644
--- a/content/blog/page10/index.html
+++ b/content/blog/page10/index.html
@@ -232,6 +232,19 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a
href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State
Unlocked: Interacting with State in Apache Flink</a></h2>
+
+ <p>29 Jan 2020
+ Seth Wiesman (<a
href="https://twitter.com/sjwiesman";>@sjwiesman</a>)</p>
+
+ <p>This post discusses the efforts of the Flink community as they relate
to state management in Apache Flink. We showcase some practical examples of how
the different features and APIs can be utilized and cover some future ideas for
new and improved ways of managing state in Apache Flink.</p>
+
+ <p><a
href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">Continue
reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a
href="/news/2020/01/15/demo-fraud-detection.html">Advanced Flink Application
Patterns Vol.1: Case Study of a Fraud Detection System</a></h2>
@@ -358,19 +371,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a
href="/2019/07/23/flink-network-stack-2.html">Flink Network Stack Vol. 2:
Monitoring, Metrics, and that Backpressure Thing</a></h2>
-
- <p>23 Jul 2019
- Nico Kruber & Piotr Nowojski </p>
-
- <p>In a previous blog post, we presented how Flink’s network stack works
from the high-level abstractions to the low-level details. This second post
discusses monitoring network-related metrics to identify backpressure or
bottlenecks in throughput and latency.</p>
-
- <p><a href="/2019/07/23/flink-network-stack-2.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -403,6 +403,16 @@
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/blog/page11/index.html b/content/blog/page11/index.html
index 34765b5e1..f8acb25b4 100644
--- a/content/blog/page11/index.html
+++ b/content/blog/page11/index.html
@@ -232,6 +232,19 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a
href="/2019/07/23/flink-network-stack-2.html">Flink Network Stack Vol. 2:
Monitoring, Metrics, and that Backpressure Thing</a></h2>
+
+ <p>23 Jul 2019
+ Nico Kruber & Piotr Nowojski </p>
+
+ <p>In a previous blog post, we presented how Flink’s network stack works
from the high-level abstractions to the low-level details. This second post
discusses monitoring network-related metrics to identify backpressure or
bottlenecks in throughput and latency.</p>
+
+ <p><a href="/2019/07/23/flink-network-stack-2.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a
href="/news/2019/07/02/release-1.8.1.html">Apache Flink 1.8.1 Released</a></h2>
@@ -359,19 +372,6 @@ for more details.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2019/03/06/ffsf-preview.html">What
to expect from Flink Forward San Francisco 2019</a></h2>
-
- <p>06 Mar 2019
- Fabian Hueske (<a href="https://twitter.com/fhueske";>@fhueske</a>)</p>
-
- <p>The third annual Flink Forward conference in San Francisco is just a
few weeks away. Let's see what Flink Forward SF 2019 has in store for the
Apache Flink and stream processing communities. This post covers some of its
highlights!</p>
-
- <p><a href="/news/2019/03/06/ffsf-preview.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -404,6 +404,16 @@ for more details.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/blog/page12/index.html b/content/blog/page12/index.html
index b0760e7fd..d542f250a 100644
--- a/content/blog/page12/index.html
+++ b/content/blog/page12/index.html
@@ -232,6 +232,19 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/news/2019/03/06/ffsf-preview.html">What
to expect from Flink Forward San Francisco 2019</a></h2>
+
+ <p>06 Mar 2019
+ Fabian Hueske (<a href="https://twitter.com/fhueske";>@fhueske</a>)</p>
+
+ <p>The third annual Flink Forward conference in San Francisco is just a
few weeks away. Let's see what Flink Forward SF 2019 has in store for the
Apache Flink and stream processing communities. This post covers some of its
highlights!</p>
+
+ <p><a href="/news/2019/03/06/ffsf-preview.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a
href="/news/2019/02/25/monitoring-best-practices.html">Monitoring Apache Flink
Applications 101</a></h2>
@@ -365,21 +378,6 @@ Please check the <a
href="https://issues.apache.org/jira/secure/ReleaseNote.jspa
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2018/10/29/release-1.5.5.html">Apache Flink 1.5.5 Released</a></h2>
-
- <p>29 Oct 2018
- </p>
-
- <p><p>The Apache Flink community released the fifth bugfix version of
the Apache Flink 1.5 series.</p>
-
-</p>
-
- <p><a href="/news/2018/10/29/release-1.5.5.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -412,6 +410,16 @@ Please check the <a
href="https://issues.apache.org/jira/secure/ReleaseNote.jspa
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/blog/page13/index.html b/content/blog/page13/index.html
index 8c80f12a8..b23b9466e 100644
--- a/content/blog/page13/index.html
+++ b/content/blog/page13/index.html
@@ -232,6 +232,21 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a
href="/news/2018/10/29/release-1.5.5.html">Apache Flink 1.5.5 Released</a></h2>
+
+ <p>29 Oct 2018
+ </p>
+
+ <p><p>The Apache Flink community released the fifth bugfix version of
the Apache Flink 1.5 series.</p>
+
+</p>
+
+ <p><a href="/news/2018/10/29/release-1.5.5.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a
href="/news/2018/09/20/release-1.6.1.html">Apache Flink 1.6.1 Released</a></h2>
@@ -367,19 +382,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a
href="/features/2018/03/01/end-to-end-exactly-once-apache-flink.html">An
Overview of End-to-End Exactly-Once Processing in Apache Flink (with Apache
Kafka, too!)</a></h2>
-
- <p>01 Mar 2018
- Piotr Nowojski (<a
href="https://twitter.com/PiotrNowojski";>@PiotrNowojski</a>) & Mike Winters
(<a href="https://twitter.com/wints";>@wints</a>)</p>
-
- <p>Flink 1.4.0 introduced a new feature that makes it possible to build
end-to-end exactly-once applications with Flink and data sources and sinks that
support transactions.</p>
-
- <p><a
href="/features/2018/03/01/end-to-end-exactly-once-apache-flink.html">Continue
reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -412,6 +414,16 @@
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/blog/page14/index.html b/content/blog/page14/index.html
index d882f1a53..f5d820553 100644
--- a/content/blog/page14/index.html
+++ b/content/blog/page14/index.html
@@ -232,6 +232,19 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a
href="/features/2018/03/01/end-to-end-exactly-once-apache-flink.html">An
Overview of End-to-End Exactly-Once Processing in Apache Flink (with Apache
Kafka, too!)</a></h2>
+
+ <p>01 Mar 2018
+ Piotr Nowojski (<a
href="https://twitter.com/PiotrNowojski";>@PiotrNowojski</a>) & Mike Winters
(<a href="https://twitter.com/wints";>@wints</a>)</p>
+
+ <p>Flink 1.4.0 introduced a new feature that makes it possible to build
end-to-end exactly-once applications with Flink and data sources and sinks that
support transactions.</p>
+
+ <p><a
href="/features/2018/03/01/end-to-end-exactly-once-apache-flink.html">Continue
reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a
href="/news/2018/02/15/release-1.4.1.html">Apache Flink 1.4.1 Released</a></h2>
@@ -366,21 +379,6 @@ what’s coming in Flink 1.4.0 as well as a preview of what
the Flink community
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2017/05/16/official-docker-image.html">Introducing Docker Images
for Apache Flink</a></h2>
-
- <p>16 May 2017 by Patrick Lucas (Data Artisans) and Ismaël Mejía
(Talend) (<a href="https://twitter.com/";>@iemejia</a>)
- </p>
-
- <p><p>For some time, the Apache Flink community has provided scripts to
build a Docker image to run Flink. Now, starting with version 1.2.1, Flink will
have a <a href="https://hub.docker.com/r/_/flink/";>Docker image</a> on the
Docker Hub. This image is maintained by the Flink community and curated by the
<a href="https://github.com/docker-library/official-images";>Docker</a> team to
ensure it meets the quality standards for container images of the Docker
community.</p>
-
-</p>
-
- <p><a href="/news/2017/05/16/official-docker-image.html">Continue
reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -413,6 +411,16 @@ what’s coming in Flink 1.4.0 as well as a preview of what
the Flink community
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/blog/page15/index.html b/content/blog/page15/index.html
index c145a330d..198312a24 100644
--- a/content/blog/page15/index.html
+++ b/content/blog/page15/index.html
@@ -232,6 +232,21 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a
href="/news/2017/05/16/official-docker-image.html">Introducing Docker Images
for Apache Flink</a></h2>
+
+ <p>16 May 2017 by Patrick Lucas (Data Artisans) and Ismaël Mejía
(Talend) (<a href="https://twitter.com/";>@iemejia</a>)
+ </p>
+
+ <p><p>For some time, the Apache Flink community has provided scripts to
build a Docker image to run Flink. Now, starting with version 1.2.1, Flink will
have a <a href="https://hub.docker.com/r/_/flink/";>Docker image</a> on the
Docker Hub. This image is maintained by the Flink community and curated by the
<a href="https://github.com/docker-library/official-images";>Docker</a> team to
ensure it meets the quality standards for container images of the Docker
community.</p>
+
+</p>
+
+ <p><a href="/news/2017/05/16/official-docker-image.html">Continue
reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a
href="/news/2017/04/26/release-1.2.1.html">Apache Flink 1.2.1 Released</a></h2>
@@ -360,21 +375,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2016/08/24/ff16-keynotes-panels.html">Flink Forward 2016:
Announcing Schedule, Keynotes, and Panel Discussion</a></h2>
-
- <p>24 Aug 2016
- </p>
-
- <p><p>An update for the Flink community: the <a
href="http://flink-forward.org/kb_day/day-1/";>Flink Forward 2016 schedule</a>
is now available online. This year's event will include 2 days of talks from
stream processing experts at Google, MapR, Alibaba, Netflix, Cloudera, and
more. Following the talks is a full day of hands-on Flink training.</p>
-
-</p>
-
- <p><a href="/news/2016/08/24/ff16-keynotes-panels.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -407,6 +407,16 @@
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/blog/page16/index.html b/content/blog/page16/index.html
index 5e0cbe88b..b8bd77ca8 100644
--- a/content/blog/page16/index.html
+++ b/content/blog/page16/index.html
@@ -232,6 +232,21 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a
href="/news/2016/08/24/ff16-keynotes-panels.html">Flink Forward 2016:
Announcing Schedule, Keynotes, and Panel Discussion</a></h2>
+
+ <p>24 Aug 2016
+ </p>
+
+ <p><p>An update for the Flink community: the <a
href="http://flink-forward.org/kb_day/day-1/";>Flink Forward 2016 schedule</a>
is now available online. This year's event will include 2 days of talks from
stream processing experts at Google, MapR, Alibaba, Netflix, Cloudera, and
more. Following the talks is a full day of hands-on Flink training.</p>
+
+</p>
+
+ <p><a href="/news/2016/08/24/ff16-keynotes-panels.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a
href="/news/2016/08/11/release-1.1.1.html">Flink 1.1.1 Released</a></h2>
@@ -364,21 +379,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2016/02/11/release-0.10.2.html">Flink 0.10.2 Released</a></h2>
-
- <p>11 Feb 2016
- </p>
-
- <p><p>Today, the Flink community released Flink version
<strong>0.10.2</strong>, the second bugfix release of the 0.10 series.</p>
-
-</p>
-
- <p><a href="/news/2016/02/11/release-0.10.2.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -411,6 +411,16 @@
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/blog/page17/index.html b/content/blog/page17/index.html
index f7a3b86d7..9b508a0aa 100644
--- a/content/blog/page17/index.html
+++ b/content/blog/page17/index.html
@@ -232,6 +232,21 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a
href="/news/2016/02/11/release-0.10.2.html">Flink 0.10.2 Released</a></h2>
+
+ <p>11 Feb 2016
+ </p>
+
+ <p><p>Today, the Flink community released Flink version
<strong>0.10.2</strong>, the second bugfix release of the 0.10 series.</p>
+
+</p>
+
+ <p><a href="/news/2016/02/11/release-0.10.2.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a
href="/news/2015/12/18/a-year-in-review.html">Flink 2015: A year in review, and
a lookout to 2016</a></h2>
@@ -368,21 +383,6 @@ vertex-centric or gather-sum-apply to Flink dataflows.</p>
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2015/06/24/announcing-apache-flink-0.9.0-release.html">Announcing
Apache Flink 0.9.0</a></h2>
-
- <p>24 Jun 2015
- </p>
-
- <p><p>The Apache Flink community is pleased to announce the availability
of the 0.9.0 release. The release is the result of many months of hard work
within the Flink community. It contains many new features and improvements
which were previewed in the 0.9.0-milestone1 release and have been polished
since then. This is the largest Flink release so far.</p>
-
-</p>
-
- <p><a
href="/news/2015/06/24/announcing-apache-flink-0.9.0-release.html">Continue
reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -415,6 +415,16 @@ vertex-centric or gather-sum-apply to Flink dataflows.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/blog/page18/index.html b/content/blog/page18/index.html
index 96bfb087a..6f127f9e8 100644
--- a/content/blog/page18/index.html
+++ b/content/blog/page18/index.html
@@ -232,6 +232,21 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a
href="/news/2015/06/24/announcing-apache-flink-0.9.0-release.html">Announcing
Apache Flink 0.9.0</a></h2>
+
+ <p>24 Jun 2015
+ </p>
+
+ <p><p>The Apache Flink community is pleased to announce the availability
of the 0.9.0 release. The release is the result of many months of hard work
within the Flink community. It contains many new features and improvements
which were previewed in the 0.9.0-milestone1 release and have been polished
since then. This is the largest Flink release so far.</p>
+
+</p>
+
+ <p><a
href="/news/2015/06/24/announcing-apache-flink-0.9.0-release.html">Continue
reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a
href="/news/2015/05/14/Community-update-April.html">April 2015 in the Flink
community</a></h2>
@@ -374,21 +389,6 @@ and offers a new API including definition of flexible
windows.</p>
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2015/01/06/december-in-flink.html">December 2014 in the Flink
community</a></h2>
-
- <p>06 Jan 2015
- </p>
-
- <p><p>This is the first blog post of a “newsletter” like series where we
give a summary of the monthly activity in the Flink community. As the Flink
project grows, this can serve as a “tl;dr” for people that are not following
the Flink dev and user mailing lists, or those that are simply overwhelmed by
the traffic.</p>
-
-</p>
-
- <p><a href="/news/2015/01/06/december-in-flink.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -421,6 +421,16 @@ and offers a new API including definition of flexible
windows.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/blog/page19/index.html b/content/blog/page19/index.html
index ae99d7d21..7e9bb04e7 100644
--- a/content/blog/page19/index.html
+++ b/content/blog/page19/index.html
@@ -232,6 +232,21 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a
href="/news/2015/01/06/december-in-flink.html">December 2014 in the Flink
community</a></h2>
+
+ <p>06 Jan 2015
+ </p>
+
+ <p><p>This is the first blog post of a “newsletter” like series where we
give a summary of the monthly activity in the Flink community. As the Flink
project grows, this can serve as a “tl;dr” for people that are not following
the Flink dev and user mailing lists, or those that are simply overwhelmed by
the traffic.</p>
+
+</p>
+
+ <p><a href="/news/2015/01/06/december-in-flink.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a
href="/news/2014/11/18/hadoop-compatibility.html">Hadoop Compatibility in
Flink</a></h2>
@@ -342,6 +357,16 @@ academic and open source project that Flink originates
from.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/blog/page2/index.html b/content/blog/page2/index.html
index 3a800502b..6498d1ac7 100644
--- a/content/blog/page2/index.html
+++ b/content/blog/page2/index.html
@@ -232,6 +232,19 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a
href="/news/2022/02/18/release-1.13.6.html">Apache Flink 1.13.6 Release
Announcement</a></h2>
+
+ <p>18 Feb 2022
+ Konstantin Knauf (<a
href="https://twitter.com/snntrable";>@snntrable</a>)</p>
+
+ <p>The Apache Flink Community is please to announce another bug fix
release for Flink 1.13.</p>
+
+ <p><a href="/news/2022/02/18/release-1.13.6.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a
href="/news/2022/01/31/release-statefun-3.2.0.html">Stateful Functions 3.2.0
Release Announcement</a></h2>
@@ -356,19 +369,6 @@ This new release brings various improvements to the
StateFun runtime, a leaner w
<hr>
- <article>
- <h2 class="blog-title"><a href="/2021/11/03/flink-backward.html">Flink
Backward - The Apache Flink Retrospective</a></h2>
-
- <p>03 Nov 2021
- Johannes Moser </p>
-
- <p>A look back at the development cycle for Flink 1.14</p>
-
- <p><a href="/2021/11/03/flink-backward.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -401,6 +401,16 @@ This new release brings various improvements to the
StateFun runtime, a leaner w
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/blog/page3/index.html b/content/blog/page3/index.html
index c3e30f676..4d4f8bafe 100644
--- a/content/blog/page3/index.html
+++ b/content/blog/page3/index.html
@@ -232,6 +232,19 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/2021/11/03/flink-backward.html">Flink
Backward - The Apache Flink Retrospective</a></h2>
+
+ <p>03 Nov 2021
+ Johannes Moser </p>
+
+ <p>A look back at the development cycle for Flink 1.14</p>
+
+ <p><a href="/2021/11/03/flink-backward.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a
href="/2021/10/26/sort-shuffle-part2.html">Sort-Based Blocking Shuffle
Implementation in Flink - Part Two</a></h2>
@@ -369,21 +382,6 @@ This new release brings various improvements to the
StateFun runtime, a leaner w
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2021/08/06/release-1.13.2.html">Apache Flink 1.13.2
Released</a></h2>
-
- <p>06 Aug 2021
- Yun Tang </p>
-
- <p><p>The Apache Flink community released the second bugfix version of
the Apache Flink 1.13 series.</p>
-
-</p>
-
- <p><a href="/news/2021/08/06/release-1.13.2.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -416,6 +414,16 @@ This new release brings various improvements to the
StateFun runtime, a leaner w
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/blog/page4/index.html b/content/blog/page4/index.html
index dc0e075ae..bc1f664cd 100644
--- a/content/blog/page4/index.html
+++ b/content/blog/page4/index.html
@@ -232,6 +232,21 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a
href="/news/2021/08/06/release-1.13.2.html">Apache Flink 1.13.2
Released</a></h2>
+
+ <p>06 Aug 2021
+ Yun Tang </p>
+
+ <p><p>The Apache Flink community released the second bugfix version of
the Apache Flink 1.13 series.</p>
+
+</p>
+
+ <p><a href="/news/2021/08/06/release-1.13.2.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a
href="/news/2021/08/06/release-1.12.5.html">Apache Flink 1.12.5
Released</a></h2>
@@ -361,21 +376,6 @@ to develop scalable, consistent, and elastic distributed
applications.</p>
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2021/03/03/release-1.12.2.html">Apache Flink 1.12.2
Released</a></h2>
-
- <p>03 Mar 2021
- Yuan Mei & Roman Khachatryan </p>
-
- <p><p>The Apache Flink community released the next bugfix version of the
Apache Flink 1.12 series.</p>
-
-</p>
-
- <p><a href="/news/2021/03/03/release-1.12.2.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -408,6 +408,16 @@ to develop scalable, consistent, and elastic distributed
applications.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/blog/page5/index.html b/content/blog/page5/index.html
index 2aa81c90e..536fa85d0 100644
--- a/content/blog/page5/index.html
+++ b/content/blog/page5/index.html
@@ -232,6 +232,21 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a
href="/news/2021/03/03/release-1.12.2.html">Apache Flink 1.12.2
Released</a></h2>
+
+ <p>03 Mar 2021
+ Yuan Mei & Roman Khachatryan </p>
+
+ <p><p>The Apache Flink community released the next bugfix version of the
Apache Flink 1.12 series.</p>
+
+</p>
+
+ <p><a href="/news/2021/03/03/release-1.12.2.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a href="/2021/02/10/native-k8s-with-ha.html">How
to natively deploy Flink on Kubernetes with High-Availability (HA)</a></h2>
@@ -357,19 +372,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2020/12/10/release-1.12.0.html">Apache Flink 1.12.0 Release
Announcement</a></h2>
-
- <p>10 Dec 2020
- Marta Paes (<a href="https://twitter.com/morsapaes";>@morsapaes</a>)
& Aljoscha Krettek (<a
href="https://twitter.com/aljoscha";>@aljoscha</a>)</p>
-
- <p>The Apache Flink community is excited to announce the release of
Flink 1.12.0! Close to 300 contributors worked on over 1k threads to bring
significant improvements to usability as well as new features to Flink users
across the whole API stack. We're particularly excited about adding efficient
batch execution to the DataStream API, Kubernetes HA as an alternative to
ZooKeeper, support for upsert mode in the Kafka SQL connector and the new
Python DataStream API! Read on for all m [...]
-
- <p><a href="/news/2020/12/10/release-1.12.0.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -402,6 +404,16 @@
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/blog/page6/index.html b/content/blog/page6/index.html
index a45350467..b434b454a 100644
--- a/content/blog/page6/index.html
+++ b/content/blog/page6/index.html
@@ -232,6 +232,19 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a
href="/news/2020/12/10/release-1.12.0.html">Apache Flink 1.12.0 Release
Announcement</a></h2>
+
+ <p>10 Dec 2020
+ Marta Paes (<a href="https://twitter.com/morsapaes";>@morsapaes</a>)
& Aljoscha Krettek (<a
href="https://twitter.com/aljoscha";>@aljoscha</a>)</p>
+
+ <p>The Apache Flink community is excited to announce the release of
Flink 1.12.0! Close to 300 contributors worked on over 1k threads to bring
significant improvements to usability as well as new features to Flink users
across the whole API stack. We're particularly excited about adding efficient
batch execution to the DataStream API, Kubernetes HA as an alternative to
ZooKeeper, support for upsert mode in the Kafka SQL connector and the new
Python DataStream API! Read on for all m [...]
+
+ <p><a href="/news/2020/12/10/release-1.12.0.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a
href="/news/2020/11/11/release-statefun-2.2.1.html">Stateful Functions 2.2.1
Release Announcement</a></h2>
@@ -361,21 +374,6 @@ as well as increased observability for operational
purposes.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/2020/08/19/statefun.html">Monitoring
and Controlling Networks of IoT Devices with Flink Stateful Functions</a></h2>
-
- <p>19 Aug 2020
- Igal Shilman (<a
href="https://twitter.com/IgalShilman";>@IgalShilman</a>)</p>
-
- <p><p>In this blog post, we’ll take a look at a class of use cases that
is a natural fit for <a
href="https://flink.apache.org/stateful-functions.html";>Flink Stateful
Functions</a>: monitoring and controlling networks of connected devices (often
called the “Internet of Things” (IoT)).</p>
-
-</p>
-
- <p><a href="/2020/08/19/statefun.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -408,6 +406,16 @@ as well as increased observability for operational
purposes.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/blog/page7/index.html b/content/blog/page7/index.html
index 8fc433313..c2331844a 100644
--- a/content/blog/page7/index.html
+++ b/content/blog/page7/index.html
@@ -232,6 +232,21 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a href="/2020/08/19/statefun.html">Monitoring
and Controlling Networks of IoT Devices with Flink Stateful Functions</a></h2>
+
+ <p>19 Aug 2020
+ Igal Shilman (<a
href="https://twitter.com/IgalShilman";>@IgalShilman</a>)</p>
+
+ <p><p>In this blog post, we’ll take a look at a class of use cases that
is a natural fit for <a
href="https://flink.apache.org/stateful-functions.html";>Flink Stateful
Functions</a>: monitoring and controlling networks of connected devices (often
called the “Internet of Things” (IoT)).</p>
+
+</p>
+
+ <p><a href="/2020/08/19/statefun.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a
href="/news/2020/08/06/external-resource.html">Accelerating your workload with
GPU and other external resources</a></h2>
@@ -360,22 +375,6 @@ illustrate this trend.</p>
<hr>
- <article>
- <h2 class="blog-title"><a
href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Flink on Zeppelin
Notebooks for Interactive Data Analysis - Part 2</a></h2>
-
- <p>23 Jun 2020
- Jeff Zhang (<a href="https://twitter.com/zjffdu";>@zjffdu</a>)</p>
-
- <p><p>In a previous post, we introduced the basics of Flink on Zeppelin
and how to do Streaming ETL. In this second part of the “Flink on Zeppelin”
series of posts, I will share how to
-perform streaming data visualization via Flink on Zeppelin and how to use
Apache Flink UDFs in Zeppelin.</p>
-
-</p>
-
- <p><a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Continue
reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -408,6 +407,16 @@ perform streaming data visualization via Flink on Zeppelin
and how to use Apache
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/blog/page8/index.html b/content/blog/page8/index.html
index 8cfec02fb..aadc1ee6c 100644
--- a/content/blog/page8/index.html
+++ b/content/blog/page8/index.html
@@ -232,6 +232,22 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a
href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Flink on Zeppelin
Notebooks for Interactive Data Analysis - Part 2</a></h2>
+
+ <p>23 Jun 2020
+ Jeff Zhang (<a href="https://twitter.com/zjffdu";>@zjffdu</a>)</p>
+
+ <p><p>In a previous post, we introduced the basics of Flink on Zeppelin
and how to do Streaming ETL. In this second part of the “Flink on Zeppelin”
series of posts, I will share how to
+perform streaming data visualization via Flink on Zeppelin and how to use
Apache Flink UDFs in Zeppelin.</p>
+
+</p>
+
+ <p><a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Continue
reading »</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a
href="/news/2020/06/15/flink-on-zeppelin-part1.html">Flink on Zeppelin
Notebooks for Interactive Data Analysis - Part 1</a></h2>
@@ -359,19 +375,6 @@ and provide a tutorial for running Streaming ETL with
Flink on Zeppelin.</p>
<hr>
- <article>
- <h2 class="blog-title"><a
href="/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python
Support for UDFs in Flink's Table API</a></h2>
-
- <p>09 Apr 2020
- Jincheng Sun (<a
href="https://twitter.com/sunjincheng121";>@sunjincheng121</a>) & Markos
Sfikas (<a href="https://twitter.com/MarkSfik";>@MarkSfik</a>)</p>
-
- <p>Flink 1.10 extends its support for Python by adding Python UDFs in
PyFlink. This post explains how UDFs work in PyFlink and gives some practical
examples of how to use UDFs in PyFlink.</p>
-
- <p><a href="/2020/04/09/pyflink-udf-support-flink.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -404,6 +407,16 @@ and provide a tutorial for running Streaming ETL with
Flink on Zeppelin.</p>
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/blog/page9/index.html b/content/blog/page9/index.html
index 2c9996f22..e92be5049 100644
--- a/content/blog/page9/index.html
+++ b/content/blog/page9/index.html
@@ -232,6 +232,19 @@
<div class="col-sm-8">
<!-- Blog posts -->
+ <article>
+ <h2 class="blog-title"><a
href="/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python
Support for UDFs in Flink's Table API</a></h2>
+
+ <p>09 Apr 2020
+ Jincheng Sun (<a
href="https://twitter.com/sunjincheng121";>@sunjincheng121</a>) & Markos
Sfikas (<a href="https://twitter.com/MarkSfik";>@MarkSfik</a>)</p>
+
+ <p>Flink 1.10 extends its support for Python by adding Python UDFs in
PyFlink. This post explains how UDFs work in PyFlink and gives some practical
examples of how to use UDFs in PyFlink.</p>
+
+ <p><a href="/2020/04/09/pyflink-udf-support-flink.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
<article>
<h2 class="blog-title"><a
href="/news/2020/04/07/release-statefun-2.0.0.html">Stateful Functions 2.0 - An
Event-driven Database on Apache Flink</a></h2>
@@ -358,19 +371,6 @@ This release marks a big milestone: Stateful Functions 2.0
is not only an API up
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State
Unlocked: Interacting with State in Apache Flink</a></h2>
-
- <p>29 Jan 2020
- Seth Wiesman (<a
href="https://twitter.com/sjwiesman";>@sjwiesman</a>)</p>
-
- <p>This post discusses the efforts of the Flink community as they relate
to state management in Apache Flink. We showcase some practical examples of how
the different features and APIs can be utilized and cover some future ideas for
new and improved ways of managing state in Apache Flink.</p>
-
- <p><a
href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">Continue
reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -403,6 +403,16 @@ This release marks a big milestone: Stateful Functions 2.0
is not only an API up
<ul id="markdown-toc">
+ <li><a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></li>
diff --git a/content/img/blog/2022-05-23-latency-part2/async-io.png
b/content/img/blog/2022-05-23-latency-part2/async-io.png
new file mode 100644
index 000000000..b1d836133
Binary files /dev/null and
b/content/img/blog/2022-05-23-latency-part2/async-io.png differ
diff --git
a/content/img/blog/2022-05-23-latency-part2/enriching-with-async-io.png
b/content/img/blog/2022-05-23-latency-part2/enriching-with-async-io.png
new file mode 100644
index 000000000..fa11bf98c
Binary files /dev/null and
b/content/img/blog/2022-05-23-latency-part2/enriching-with-async-io.png differ
diff --git a/content/img/blog/2022-05-23-latency-part2/spread-work.png
b/content/img/blog/2022-05-23-latency-part2/spread-work.png
new file mode 100644
index 000000000..e0c40c5d0
Binary files /dev/null and
b/content/img/blog/2022-05-23-latency-part2/spread-work.png differ
diff --git a/content/index.html b/content/index.html
index 69da5427f..eb2ef30f8 100644
--- a/content/index.html
+++ b/content/index.html
@@ -397,6 +397,9 @@
<dl>
+ <dt> <a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></dt>
+ <dd>This multi-part series of blog post presents a collection of
low-latency techniques in Flink. Following with part one, Part two continues
with a few more techniques that optimize latency directly.</dd>
+
<dt> <a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></dt>
<dd>This multi-part series of blog post presents a collection of
low-latency techniques in Flink. Part one starts with types of latency in Flink
and the way we measure the end-to-end latency, followed by a few techniques
that optimize latency directly.</dd>
@@ -411,9 +414,6 @@
<dt> <a href="/2022/05/06/pyflink-1.15-thread-mode.html">Exploring the
thread mode in PyFlink</a></dt>
<dd>Flink 1.15 introduced a new Runtime Execution Mode named 'thread'
mode in PyFlink. This post explains how it works and when to use it.</dd>
-
- <dt> <a href="/2022/05/06/restore-modes.html">Improvements to Flink
operations: Snapshots Ownership and Savepoint Formats</a></dt>
- <dd>This post will outline the journey of improving snapshotting in
past releases and the upcoming improvements in Flink 1.15, which includes
making it possible to take savepoints in the native state backend specific
format as well as clarifying snapshots ownership.</dd>
</dl>
diff --git a/content/zh/index.html b/content/zh/index.html
index e6d695f5e..39afbfe35 100644
--- a/content/zh/index.html
+++ b/content/zh/index.html
@@ -394,6 +394,9 @@
<dl>
+ <dt> <a href="/2022/05/23/latency-part2.html">Getting into Low-Latency
Gears with Apache Flink - Part Two</a></dt>
+ <dd>This multi-part series of blog post presents a collection of
low-latency techniques in Flink. Following with part one, Part two continues
with a few more techniques that optimize latency directly.</dd>
+
<dt> <a href="/2022/05/18/latency-part1.html">Getting into Low-Latency
Gears with Apache Flink - Part One</a></dt>
<dd>This multi-part series of blog post presents a collection of
low-latency techniques in Flink. Part one starts with types of latency in Flink
and the way we measure the end-to-end latency, followed by a few techniques
that optimize latency directly.</dd>
@@ -408,9 +411,6 @@
<dt> <a href="/2022/05/06/pyflink-1.15-thread-mode.html">Exploring the
thread mode in PyFlink</a></dt>
<dd>Flink 1.15 introduced a new Runtime Execution Mode named 'thread'
mode in PyFlink. This post explains how it works and when to use it.</dd>
-
- <dt> <a href="/2022/05/06/restore-modes.html">Improvements to Flink
operations: Snapshots Ownership and Savepoint Formats</a></dt>
- <dd>This post will outline the journey of improving snapshotting in
past releases and the upcoming improvements in Flink 1.15, which includes
making it possible to take savepoints in the native state backend specific
format as well as clarifying snapshots ownership.</dd>
</dl>
