This is an automated email from the ASF dual-hosted git repository. rmetzger pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit a47c0cff9575dd447a1c4d275de3428ecca49269 Author: Robert Metzger <[email protected]> AuthorDate: Wed May 22 15:24:00 2019 +0200 rebuild site --- content/2019/05/19/state-ttl.html | 335 ++++++++++++++++++++++++++++++++++++++ content/blog/index.html | 36 ++-- content/blog/page2/index.html | 38 +++-- content/blog/page3/index.html | 42 +++-- content/blog/page4/index.html | 42 +++-- content/blog/page5/index.html | 40 +++-- content/blog/page6/index.html | 40 +++-- content/blog/page7/index.html | 40 +++-- content/blog/page8/index.html | 40 +++-- content/blog/page9/index.html | 25 +++ content/index.html | 6 +- content/zh/index.html | 6 +- 12 files changed, 564 insertions(+), 126 deletions(-) diff --git a/content/2019/05/19/state-ttl.html b/content/2019/05/19/state-ttl.html new file mode 100644 index 0000000..3c0661a --- /dev/null +++ b/content/2019/05/19/state-ttl.html @@ -0,0 +1,335 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> + <title>Apache Flink: State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</title> + <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> + <link rel="icon" href="/favicon.ico" type="image/x-icon"> + + <!-- Bootstrap --> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css";> + <link rel="stylesheet" href="/css/flink.css"> + <link rel="stylesheet" href="/css/syntax.css"> + + <!-- Blog RSS feed --> + <link href="/blog/feed.xml" rel="alternate" type="application/rss+xml" title="Apache Flink Blog: RSS feed" /> + + <!-- jQuery (necessary for Bootstrap's JavaScript plugins) --> + <!-- We need to load Jquery in the header for custom google analytics event tracking--> + <script src="/js/jquery.min.js"></script> + + <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> + <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> + <!--[if lt IE 9]> + <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js";></script> + <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js";></script> + <![endif]--> + </head> + <body> + + + <!-- Main content. --> + <div class="container"> + <div class="row"> + + + <div id="sidebar" class="col-sm-3"> + + +<!-- Top navbar. --> + <nav class="navbar navbar-default"> + <!-- The logo. --> + <div class="navbar-header"> + <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <div class="navbar-logo"> + <a href="/"> + <img alt="Apache Flink" src="/img/flink-header-logo.svg" width="147px" height="73px"> + </a> + </div> + </div><!-- /.navbar-header --> + + <!-- The navigation links. --> + <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1"> + <ul class="nav navbar-nav navbar-main"> + + <!-- First menu section explains visitors what Flink is --> + + <!-- What is Stream Processing? --> + <!-- + <li><a href="/streamprocessing1.html">What is Stream Processing?</a></li> + --> + + <!-- What is Flink? --> + <li><a href="/flink-architecture.html">What is Apache Flink?</a></li> + + <!-- Use cases --> + <li><a href="/usecases.html">Use Cases</a></li> + + <!-- Powered by --> + <li><a href="/poweredby.html">Powered By</a></li> + + <!-- FAQ --> + <li><a href="/faq.html">FAQ</a></li> + + + <!-- Second menu section aims to support Flink users --> + + <!-- Downloads --> + <li><a href="/downloads.html">Downloads</a></li> + + <!-- Quickstart --> + <li> + <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/quickstart/setup_quickstart.html"; target="_blank">Tutorials <small><span class="glyphicon glyphicon-new-window"></span></small></a> + </li> + + <!-- Documentation --> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8"; target="_blank">1.8 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + <li><a href="https://ci.apache.org/projects/flink/flink-docs-master"; target="_blank">1.9 (Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + </ul> + </li> + + <!-- getting help --> + <li><a href="/gettinghelp.html">Getting Help</a></li> + + <!-- Blog --> + <li><a href="/blog/"><b>Flink Blog</b></a></li> + + + + <!-- Third menu section aim to support community and contributors --> + + <!-- Community --> + <li><a href="/community.html">Community & Project Info</a></li> + + <!-- Roadmap --> + <li><a href="/roadmap.html">Roadmap</a></li> + + <!-- Contribute --> + <li><a href="/how-to-contribute.html">How to Contribute</a></li> + + <!-- GitHub --> + <li> + <a href="https://github.com/apache/flink"; target="_blank">Flink on GitHub <small><span class="glyphicon glyphicon-new-window"></span></small></a> + </li> + + + + <!-- Language Switcher --> + <li> + + + <a href="/zh/2019/05/19/state-ttl.html">中文版</a> + + + </li> + + </ul> + + <ul class="nav navbar-nav navbar-bottom"> + <hr /> + + <!-- Twitter --> + <li><a href="https://twitter.com/apacheflink"; target="_blank">@ApacheFlink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + + <!-- Visualizer --> + <li class=" hidden-md hidden-sm"><a href="/visualizer/" target="_blank">Plan Visualizer <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + + </ul> + </div><!-- /.navbar-collapse --> + </nav> + + </div> + <div class="col-sm-9"> + <div class="row-fluid"> + <div class="col-sm-12"> + <div class="row"> + <h1>State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</h1> + + <article> + <p>19 May 2019 Fabian Hueske (<a href="https://twitter.com/fhueske";>@fhueske</a>) & Andrey Zagrebin </p> + +<p>A common requirement for many stateful streaming applications is to automatically cleanup application state for effective management of your state size, or to control how long the application state can be accessed (e.g. due to legal regulations like the GDPR). The state time-to-live (TTL) feature was initiated in Flink 1.6.0 and enabled application state cleanup and efficient state size management in Apache Flink.</p> + +<p>In this post, we motivate the State TTL feature and discuss its use cases. Moreover, we show how to use and configure it. We explain how Flink internally manages state with TTL and present some exciting additions to the feature in Flink 1.8.0. The blog post concludes with an outlook on future improvements and extensions.</p> + +<h1 id="the-transient-nature-of-state">The Transient Nature of State</h1> + +<p>There are two major reasons why state should be maintained only for a limited time. For example, let’s imagine a Flink application that ingests a stream of user login events and stores for each user the time of the last login to improve the experience of frequent visitors.</p> + +<ul> + <li> + <p><strong>Controlling the size of state.</strong> +Being able to efficiently manage an ever-growing state size is a primary use case for state TTL. Oftentimes, data needs to be persisted temporarily while there is some user activity around it, e.g. web sessions. When the activity ends there is no longer interest in that data while it still occupies storage. Flink 1.8.0 introduces background cleanup of old state based on TTL that makes the eviction of no-longer-necessary data frictionless. Previously, the application developer had to take [...] + </li> + <li> + <p><strong>Complying with data protection and sensitive data requirements.</strong> +Recent developments around data privacy regulations, such as the General Data Protection Regulation (GDPR) introduced by the European Union, make compliance with such data requirements or treating sensitive data a top priority for many use cases and applications. An example of such use cases includes applications that require keeping data for a specific timeframe and preventing access to it thereafter. This is a common challenge for companies providing short-term services to their custom [...] + </li> +</ul> + +<p>Both requirements can be addressed by a feature that periodically, yet continuously, removes the state for a key once it becomes unnecessary or unimportant and there is no requirement to keep it in storage any more.</p> + +<h1 id="state-ttl-for-continuous-cleanup-of-application-state">State TTL for continuous cleanup of application state</h1> + +<p>The 1.6.0 release of Apache Flink introduced the State TTL feature. It enabled developers of stream processing applications to configure the state of operators to expire and be cleaned up after a defined timeout (time-to-live). In Flink 1.8.0 the feature was extended, including continuous cleanup of old entries for both the RocksDB and the heap state backends (FSStateBackend and MemoryStateBackend), enabling a continuous cleanup process of old entries (according to the TTL setting).</p> + +<p>In Flink’s DataStream API, application state is defined by a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/state/state.html#using-managed-keyed-state";>state descriptor</a>. State TTL is configured by passing a <code>StateTtlConfiguration</code> object to a state descriptor. The following Java example shows how to create a state TTL configuration and provide it to the state descriptor that holds the last login time of a user as a <code>Long</code> value:</p> + +<div class="highlight"><pre><code class="language-java"><span class="kn">import</span> <span class="nn">org.apache.flink.api.common.state.StateTtlConfig</span><span class="o">;</span> +<span class="kn">import</span> <span class="nn">org.apache.flink.api.common.time.Time</span><span class="o">;</span> +<span class="kn">import</span> <span class="nn">org.apache.flink.api.common.state.ValueStateDescriptor</span><span class="o">;</span> + +<span class="n">StateTtlConfig</span> <span class="n">ttlConfig</span> <span class="o">=</span> <span class="n">StateTtlConfig</span> + <span class="o">.</span><span class="na">newBuilder</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">days</span><span class="o">(</span><span class="mi">7</span><span class="o">))</span> + <span class="o">.</span><span class="na">setUpdateType</span><span class="o">(</span><span class="n">StateTtlConfig</span><span class="o">.</span><span class="na">UpdateType</span><span class="o">.</span><span class="na">OnCreateAndWrite</span><span class="o">)</span> + <span class="o">.</span><span class="na">setStateVisibility</span><span class="o">(</span><span class="n">StateTtlConfig</span><span class="o">.</span><span class="na">StateVisibility</span><span class="o">.</span><span class="na">NeverReturnExpired</span><span class="o">)</span> + <span class="o">.</span><span class="na">build</span><span class="o">();</span> + +<span class="n">ValueStateDescriptor</span><span class="o"><</span><span class="n">Long</span><span class="o">></span> <span class="n">lastUserLogin</span> <span class="o">=</span> + <span class="k">new</span> <span class="n">ValueStateDescriptor</span><span class="o"><>(</span><span class="s">"lastUserLogin"</span><span class="o">,</span> <span class="n">Long</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> + +<span class="n">lastUserLogin</span><span class="o">.</span><span class="na">enableTimeToLive</span><span class="o">(</span><span class="n">ttlConfig</span><span class="o">);</span></code></pre></div> + +<p>Flink provides multiple options to configure the behavior of the state TTL functionality.</p> + +<ul> + <li> + <p><strong>When is the Time-to-Live reset?</strong> +By default, the expiration time of a state entry is updated when the state is modified. Optionally, it can also be updated on read access at the cost of an additional write operation to update the timestamp.</p> + </li> + <li> + <p><strong>Can the expired state be accessed one last time?</strong> +State TTL employs a lazy strategy to clean up expired state. This can lead to the situation that an application attempts to read state which is expired but hasn’t been removed yet. You can configure whether such a read request returns the expired state or not. In either case, the expired state is immediately removed afterwards. While the option of returning expired state favors data availability, not returning expired state can be required for data protection regulations.</p> + </li> + <li> + <p><strong>Which time semantics are used for the Time-to-Live timers?</strong> +With Flink 1.8.0, users can only define a state TTL in terms of processing time. The support for event time is planned for future Apache Flink releases.</p> + </li> +</ul> + +<p>You can read more about how to use state TTL in the <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#state-time-to-live-ttl";>Apache Flink documentation</a>.</p> + +<p>Internally, the State TTL feature is implemented by storing an additional timestamp of the last relevant state access, along with the actual state value. While this approach adds some storage overhead, it allows Flink to check for the expired state during state access, checkpointing, recovery, or dedicated storage cleanup procedures.</p> + +<h1 id="taking-out-the-garbage">“Taking out the Garbage”</h1> + +<p>When a state object is accessed in a read operation, Flink will check its timestamp and clear the state if it is expired (depending on the configured state visibility, the expired state is returned or not). Due to this lazy removal, expired state that is never accessed again will forever occupy storage space unless it is garbage collected.</p> + +<p>So how can the expired state be removed without the application logic explicitly taking care of it? In general, there are different possible strategies to remove it in the background.</p> + +<h2 id="keep-full-state-snapshots-clean">Keep full state snapshots clean</h2> + +<p>Flink 1.6.0 already supported automatic eviction of the expired state when a full snapshot for a checkpoint or savepoint is taken. Note that state eviction is not applied for incremental checkpoints. State eviction on full snapshots must be explicitly enabled as shown in the following example:</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">StateTtlConfig</span> <span class="n">ttlConfig</span> <span class="o">=</span> <span class="n">StateTtlConfig</span> + <span class="o">.</span><span class="na">newBuilder</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">days</span><span class="o">(</span><span class="mi">7</span><span class="o">))</span> + <span class="o">.</span><span class="na">cleanupFullSnapshot</span><span class="o">()</span> + <span class="o">.</span><span class="na">build</span><span class="o">();</span></code></pre></div> + +<p>The local storage stays untouched but the size of the stored snapshot is reduced. The local state of an operator will only be cleaned up when the operator reloads its state from a snapshot, i.e. in case of recovery or when starting from a savepoint.</p> + +<p>Due to these limitations, applications still need to actively remove state after it expired in Flink 1.6.0. To improve the user experience, Flink 1.8.0 introduces two more autonomous cleanup strategies, one for each of Flink’s two state backend types. We describe them below.</p> + +<h2 id="incremental-cleanup-in-heap-state-backends">Incremental cleanup in Heap state backends</h2> + +<p>This approach is specific to the Heap state backends (FSStateBackend and MemoryStateBackend). The idea is that the storage backend keeps a lazy global iterator over all state entries. Certain events, for instance state access, trigger an incremental cleanup. Every time an incremental cleanup is triggered, the iterator is advanced. The traversed state entries are checked and expired once are removed. The following code example shows how to enable incremental cleanup:</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">StateTtlConfig</span> <span class="n">ttlConfig</span> <span class="o">=</span> <span class="n">StateTtlConfig</span> + <span class="o">.</span><span class="na">newBuilder</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">days</span><span class="o">(</span><span class="mi">7</span><span class="o">))</span> + <span class="c1">// check 10 keys for every state access</span> + <span class="o">.</span><span class="na">cleanupIncrementally</span><span class="o">(</span><span class="mi">10</span><span class="o">,</span> <span class="kc">false</span><span class="o">)</span> + <span class="o">.</span><span class="na">build</span><span class="o">();</span></code></pre></div> + +<p>If enabled, every state access triggers a cleanup step. For every clean up step, a certain number of state entries are checked for expiration. There are two tuning parameters. The first defines the number of state entries to check for each cleanup step. The second parameter is a flag to trigger a cleanup step after each processed record, additionally to each state access.</p> + +<p>There are two important caveats about this approach: +* The first one is that the time spent for the incremental cleanup increases the record processing latency. +* The second one should be practically negligible but still worth mentioning: if no state is accessed or no records are processed, expired state won’t be removed.</p> + +<h2 id="rocksdb-background-compaction-to-filter-out-expired-state">RocksDB background compaction to filter out expired state</h2> + +<p>If your application uses the RocksDB state backend, you can enable another cleanup strategy which is based on a Flink specific compaction filter. RocksDB periodically runs asynchronous compactions to merge state updates and reduce storage. The Flink compaction filter checks the expiration timestamp of state entries with TTL and discards all expired values.</p> + +<p>The first step to activate this feature is to configure the RocksDB state backend by setting the following Flink configuration option: <code>state.backend.rocksdb.ttl.compaction.filter.enabled</code>. Once the RocksDB state backend is configured, the compaction cleanup strategy is enabled for a state as shown in the following code example:</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">StateTtlConfig</span> <span class="n">ttlConfig</span> <span class="o">=</span> <span class="n">StateTtlConfig</span> + <span class="o">.</span><span class="na">newBuilder</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">days</span><span class="o">(</span><span class="mi">7</span><span class="o">))</span> + <span class="o">.</span><span class="na">cleanupInRocksdbCompactFilter</span><span class="o">()</span> + <span class="o">.</span><span class="na">build</span><span class="o">();</span></code></pre></div> +<p>Keep in mind that calling the Flink TTL filter slows down the RocksDB compaction.</p> + +<h2 id="eager-state-cleanup-with-timers">Eager State Cleanup with Timers</h2> + +<p>Another way to manually cleanup state is based on Flink timers. This is an idea that the community is currently evaluating for future releases. With this approach, a cleanup timer is registered for every state access. This approach is more predictable because state is eagerly removed as soon as it expires. However, it is more expensive because the timers consume storage along with the original state.</p> + +<h1 id="future-work">Future work</h1> + +<p>Apart from including the timer-based cleanup strategy, mentioned above, the Flink community has plans to further improve the state TTL feature. The possible improvements include adding support of TTL for event time scale (only processing time is supported at the moment) and enabling State TTL for queryable state.</p> + +<p>We encourage you to join the conversation and share your thoughts and ideas in the <a href="https://issues.apache.org/jira/projects/FLINK/summary";>Apache Flink JIRA board</a> or by subscribing to the Apache Flink dev mailing list. Feedback or suggestions are always appreciated and we look forward to hearing your thoughts on the Flink mailing lists.</p> + +<h1 id="summary">Summary</h1> + +<p>Time-based state access restrictions and controlling the size of application state are common challenges in the world of stateful stream processing. Flink’s 1.8.0 release significantly improves the State TTL feature by adding support for continuous background cleanup of expired state objects. The new clean up mechanisms relieve you from manually implementing state cleanup. They are also more efficient due to their lazy nature. State TTL gives you control over the size of your applicat [...] + + </article> + </div> + + <div class="row"> + <div id="disqus_thread"></div> + <script type="text/javascript"> + /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */ + var disqus_shortname = 'stratosphere-eu'; // required: replace example with your forum shortname + + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); + </script> + </div> + </div> +</div> + </div> + </div> + + <hr /> + + <div class="row"> + <div class="footer text-center col-sm-12"> + <p>Copyright © 2014-2019 <a href="http://apache.org";>The Apache Software Foundation</a>. All Rights Reserved.</p> + <p>Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.</p> + <p><a href="/privacy-policy.html">Privacy Policy</a> · <a href="/blog/feed.xml">RSS feed</a></p> + </div> + </div> + </div><!-- /.container --> + + <!-- Include all compiled plugins (below), or include individual files as needed --> + <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js";></script> + <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery.matchHeight/0.7.0/jquery.matchHeight-min.js";></script> + <script src="/js/codetabs.js"></script> + <script src="/js/stickysidebar.js"></script> + + <!-- Google Analytics --> + <script> + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + + ga('create', 'UA-52545728-1', 'auto'); + ga('send', 'pageview'); + </script> + </body> +</html> diff --git a/content/blog/index.html b/content/blog/index.html index 70a446b..9bb1c0e 100644 --- a/content/blog/index.html +++ b/content/blog/index.html @@ -159,6 +159,19 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/2019/05/19/state-ttl.html">State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</a></h2> + + <p>19 May 2019 + Fabian Hueske (<a href="https://twitter.com/fhueske";>@fhueske</a>) & Andrey Zagrebin </p> + + <p>A common requirement for many stateful streaming applications is to automatically cleanup application state for effective management of your state size, or to control how long the application state can be accessed. State TTL enables application state cleanup and efficient state size management in Apache Flink</p> + + <p><a href="/2019/05/19/state-ttl.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/2019/05/14/temporal-tables.html">Flux capacitor, huh? Temporal Tables and Joins in Streaming SQL</a></h2> <p>14 May 2019 @@ -287,19 +300,6 @@ for more details.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2019/02/13/unified-batch-streaming-blink.html">Batch as a Special Case of Streaming and Alibaba's contribution of Blink</a></h2> - - <p>13 Feb 2019 - Stephan Ewen (<a href="https://twitter.com/stephanewen";>@stephanewen</a>), Fabian Hueske (<a href="https://twitter.com/fhueske";>@fhueske</a>), & Xiaowei Jiang (<a href="https://twitter.com/XiaoweiJ";>@XiaoweiJ</a>)</p> - - <p>A few weeks ago, Alibaba contributed its Flink-fork 'Blink' back to Apache Flink. In this blog post we discuss how Blink's features will help the Flink community to make a big step towards its vision to build a truly unified system for stream and batch processing.</p> - - <p><a href="/news/2019/02/13/unified-batch-streaming-blink.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -332,6 +332,16 @@ for more details.</p> <ul id="markdown-toc"> + <li><a href="/2019/05/19/state-ttl.html">State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</a></li> + + + + + + + + + <li><a href="/2019/05/14/temporal-tables.html">Flux capacitor, huh? Temporal Tables and Joins in Streaming SQL</a></li> diff --git a/content/blog/page2/index.html b/content/blog/page2/index.html index c221556..b7e478e 100644 --- a/content/blog/page2/index.html +++ b/content/blog/page2/index.html @@ -159,6 +159,19 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2019/02/13/unified-batch-streaming-blink.html">Batch as a Special Case of Streaming and Alibaba's contribution of Blink</a></h2> + + <p>13 Feb 2019 + Stephan Ewen (<a href="https://twitter.com/stephanewen";>@stephanewen</a>), Fabian Hueske (<a href="https://twitter.com/fhueske";>@fhueske</a>), & Xiaowei Jiang (<a href="https://twitter.com/XiaoweiJ";>@XiaoweiJ</a>)</p> + + <p>A few weeks ago, Alibaba contributed its Flink-fork 'Blink' back to Apache Flink. In this blog post we discuss how Blink's features will help the Flink community to make a big step towards its vision to build a truly unified system for stream and batch processing.</p> + + <p><a href="/news/2019/02/13/unified-batch-streaming-blink.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2018/12/26/release-1.5.6.html">Apache Flink 1.5.6 Released</a></h2> <p>26 Dec 2018 @@ -295,21 +308,6 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa <hr> - <article> - <h2 class="blog-title"><a href="/news/2018/08/09/release-1.6.0.html">Apache Flink 1.6.0 Release Announcement</a></h2> - - <p>09 Aug 2018 - Till Rohrmann (<a href="https://twitter.com/stsffap";>@stsffap</a>)</p> - - <p><p>The Apache Flink community is proud to announce the 1.6.0 release. Over the past 2 months, the Flink community has worked hard to resolve more than 360 issues. Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12342760";>complete changelog</a> for more details.</p> - -</p> - - <p><a href="/news/2018/08/09/release-1.6.0.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -342,6 +340,16 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa <ul id="markdown-toc"> + <li><a href="/2019/05/19/state-ttl.html">State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</a></li> + + + + + + + + + <li><a href="/2019/05/14/temporal-tables.html">Flux capacitor, huh? Temporal Tables and Joins in Streaming SQL</a></li> diff --git a/content/blog/page3/index.html b/content/blog/page3/index.html index e5d3b10..f33add9 100644 --- a/content/blog/page3/index.html +++ b/content/blog/page3/index.html @@ -159,6 +159,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2018/08/09/release-1.6.0.html">Apache Flink 1.6.0 Release Announcement</a></h2> + + <p>09 Aug 2018 + Till Rohrmann (<a href="https://twitter.com/stsffap";>@stsffap</a>)</p> + + <p><p>The Apache Flink community is proud to announce the 1.6.0 release. Over the past 2 months, the Flink community has worked hard to resolve more than 360 issues. Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12342760";>complete changelog</a> for more details.</p> + +</p> + + <p><a href="/news/2018/08/09/release-1.6.0.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2018/07/31/release-1.5.2.html">Apache Flink 1.5.2 Released</a></h2> <p>31 Jul 2018 @@ -287,23 +302,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2017/12/12/release-1.4.0.html">Apache Flink 1.4.0 Release Announcement</a></h2> - - <p>12 Dec 2017 - Aljoscha Krettek (<a href="https://twitter.com/aljoscha";>@aljoscha</a>) & Mike Winters (<a href="https://twitter.com/wints";>@wints</a>)</p> - - <p><p>The Apache Flink community is pleased to announce the 1.4.0 release. Over the past 5 months, the -Flink community has been working hard to resolve more than 900 issues. See the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12340533";>complete changelog</a> -for more detail.</p> - -</p> - - <p><a href="/news/2017/12/12/release-1.4.0.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -336,6 +334,16 @@ for more detail.</p> <ul id="markdown-toc"> + <li><a href="/2019/05/19/state-ttl.html">State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</a></li> + + + + + + + + + <li><a href="/2019/05/14/temporal-tables.html">Flux capacitor, huh? Temporal Tables and Joins in Streaming SQL</a></li> diff --git a/content/blog/page4/index.html b/content/blog/page4/index.html index e326d8d..ea41573 100644 --- a/content/blog/page4/index.html +++ b/content/blog/page4/index.html @@ -159,6 +159,23 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2017/12/12/release-1.4.0.html">Apache Flink 1.4.0 Release Announcement</a></h2> + + <p>12 Dec 2017 + Aljoscha Krettek (<a href="https://twitter.com/aljoscha";>@aljoscha</a>) & Mike Winters (<a href="https://twitter.com/wints";>@wints</a>)</p> + + <p><p>The Apache Flink community is pleased to announce the 1.4.0 release. Over the past 5 months, the +Flink community has been working hard to resolve more than 900 issues. See the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12340533";>complete changelog</a> +for more detail.</p> + +</p> + + <p><a href="/news/2017/12/12/release-1.4.0.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2017/11/22/release-1.4-and-1.5-timeline.html">Looking Ahead to Apache Flink 1.4.0 and 1.5.0</a></h2> <p>22 Nov 2017 @@ -291,21 +308,6 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community <hr> - <article> - <h2 class="blog-title"><a href="/news/2017/03/23/release-1.1.5.html">Apache Flink 1.1.5 Released</a></h2> - - <p>23 Mar 2017 - </p> - - <p><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.1 series.</p> - -</p> - - <p><a href="/news/2017/03/23/release-1.1.5.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -338,6 +340,16 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community <ul id="markdown-toc"> + <li><a href="/2019/05/19/state-ttl.html">State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</a></li> + + + + + + + + + <li><a href="/2019/05/14/temporal-tables.html">Flux capacitor, huh? Temporal Tables and Joins in Streaming SQL</a></li> diff --git a/content/blog/page5/index.html b/content/blog/page5/index.html index 2571139..6f522a3 100644 --- a/content/blog/page5/index.html +++ b/content/blog/page5/index.html @@ -159,6 +159,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2017/03/23/release-1.1.5.html">Apache Flink 1.1.5 Released</a></h2> + + <p>23 Mar 2017 + </p> + + <p><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.1 series.</p> + +</p> + + <p><a href="/news/2017/03/23/release-1.1.5.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2017/02/06/release-1.2.0.html">Announcing Apache Flink 1.2.0</a></h2> <p>06 Feb 2017 by Robert Metzger @@ -288,21 +303,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2016/05/11/release-1.0.3.html">Flink 1.0.3 Released</a></h2> - - <p>11 May 2016 - </p> - - <p><p>Today, the Flink community released Flink version <strong>1.0.3</strong>, the third bugfix release of the 1.0 series.</p> - -</p> - - <p><a href="/news/2016/05/11/release-1.0.3.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -335,6 +335,16 @@ <ul id="markdown-toc"> + <li><a href="/2019/05/19/state-ttl.html">State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</a></li> + + + + + + + + + <li><a href="/2019/05/14/temporal-tables.html">Flux capacitor, huh? Temporal Tables and Joins in Streaming SQL</a></li> diff --git a/content/blog/page6/index.html b/content/blog/page6/index.html index a7f5054..77f877f 100644 --- a/content/blog/page6/index.html +++ b/content/blog/page6/index.html @@ -159,6 +159,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2016/05/11/release-1.0.3.html">Flink 1.0.3 Released</a></h2> + + <p>11 May 2016 + </p> + + <p><p>Today, the Flink community released Flink version <strong>1.0.3</strong>, the third bugfix release of the 1.0 series.</p> + +</p> + + <p><a href="/news/2016/05/11/release-1.0.3.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2016/04/22/release-1.0.2.html">Flink 1.0.2 Released</a></h2> <p>22 Apr 2016 @@ -286,21 +301,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2015/11/27/release-0.10.1.html">Flink 0.10.1 released</a></h2> - - <p>27 Nov 2015 - </p> - - <p><p>Today, the Flink community released the first bugfix release of the 0.10 series of Flink.</p> - -</p> - - <p><a href="/news/2015/11/27/release-0.10.1.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -333,6 +333,16 @@ <ul id="markdown-toc"> + <li><a href="/2019/05/19/state-ttl.html">State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</a></li> + + + + + + + + + <li><a href="/2019/05/14/temporal-tables.html">Flux capacitor, huh? Temporal Tables and Joins in Streaming SQL</a></li> diff --git a/content/blog/page7/index.html b/content/blog/page7/index.html index 0ce07c5..f552180 100644 --- a/content/blog/page7/index.html +++ b/content/blog/page7/index.html @@ -159,6 +159,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2015/11/27/release-0.10.1.html">Flink 0.10.1 released</a></h2> + + <p>27 Nov 2015 + </p> + + <p><p>Today, the Flink community released the first bugfix release of the 0.10 series of Flink.</p> + +</p> + + <p><a href="/news/2015/11/27/release-0.10.1.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2015/11/16/release-0.10.0.html">Announcing Apache Flink 0.10.0</a></h2> <p>16 Nov 2015 @@ -301,21 +316,6 @@ release is a preview release that contains known issues.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2015/04/07/march-in-flink.html">March 2015 in the Flink community</a></h2> - - <p>07 Apr 2015 - </p> - - <p><p>March has been a busy month in the Flink community.</p> - -</p> - - <p><a href="/news/2015/04/07/march-in-flink.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -348,6 +348,16 @@ release is a preview release that contains known issues.</p> <ul id="markdown-toc"> + <li><a href="/2019/05/19/state-ttl.html">State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</a></li> + + + + + + + + + <li><a href="/2019/05/14/temporal-tables.html">Flux capacitor, huh? Temporal Tables and Joins in Streaming SQL</a></li> diff --git a/content/blog/page8/index.html b/content/blog/page8/index.html index b9984bb..a39b497 100644 --- a/content/blog/page8/index.html +++ b/content/blog/page8/index.html @@ -159,6 +159,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2015/04/07/march-in-flink.html">March 2015 in the Flink community</a></h2> + + <p>07 Apr 2015 + </p> + + <p><p>March has been a busy month in the Flink community.</p> + +</p> + + <p><a href="/news/2015/04/07/march-in-flink.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html">Peeking into Apache Flink's Engine Room</a></h2> <p>13 Mar 2015 by Fabian Hüske (<a href="https://twitter.com/";>@fhueske</a>) @@ -298,21 +313,6 @@ and offers a new API including definition of flexible windows.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2014/09/26/release-0.6.1.html">Apache Flink 0.6.1 available</a></h2> - - <p>26 Sep 2014 - </p> - - <p><p>We are happy to announce the availability of Flink 0.6.1.</p> - -</p> - - <p><a href="/news/2014/09/26/release-0.6.1.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -345,6 +345,16 @@ and offers a new API including definition of flexible windows.</p> <ul id="markdown-toc"> + <li><a href="/2019/05/19/state-ttl.html">State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</a></li> + + + + + + + + + <li><a href="/2019/05/14/temporal-tables.html">Flux capacitor, huh? Temporal Tables and Joins in Streaming SQL</a></li> diff --git a/content/blog/page9/index.html b/content/blog/page9/index.html index a28fdbf..2c9f9f0 100644 --- a/content/blog/page9/index.html +++ b/content/blog/page9/index.html @@ -159,6 +159,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2014/09/26/release-0.6.1.html">Apache Flink 0.6.1 available</a></h2> + + <p>26 Sep 2014 + </p> + + <p><p>We are happy to announce the availability of Flink 0.6.1.</p> + +</p> + + <p><a href="/news/2014/09/26/release-0.6.1.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2014/08/26/release-0.6.html">Apache Flink 0.6 available</a></h2> <p>26 Aug 2014 @@ -208,6 +223,16 @@ academic and open source project that Flink originates from.</p> <ul id="markdown-toc"> + <li><a href="/2019/05/19/state-ttl.html">State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</a></li> + + + + + + + + + <li><a href="/2019/05/14/temporal-tables.html">Flux capacitor, huh? Temporal Tables and Joins in Streaming SQL</a></li> diff --git a/content/index.html b/content/index.html index 4c7ab41..1bdf6e1 100644 --- a/content/index.html +++ b/content/index.html @@ -448,6 +448,9 @@ <dl> + <dt> <a href="/2019/05/19/state-ttl.html">State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</a></dt> + <dd>A common requirement for many stateful streaming applications is to automatically cleanup application state for effective management of your state size, or to control how long the application state can be accessed. State TTL enables application state cleanup and efficient state size management in Apache Flink</dd> + <dt> <a href="/2019/05/14/temporal-tables.html">Flux capacitor, huh? Temporal Tables and Joins in Streaming SQL</a></dt> <dd>Apache Flink natively supports temporal table joins since the 1.7 release for straightforward temporal data handling. In this blog post, we provide an overview of how this new concept can be leveraged for effective point-in-time analysis in streaming scenarios.</dd> @@ -467,9 +470,6 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa for more details.</p> </dd> - - <dt> <a href="/features/2019/03/11/prometheus-monitoring.html">Flink and Prometheus: Cloud-native monitoring of streaming applications</a></dt> - <dd>This blog post describes how developers can leverage Apache Flink's built-in metrics system together with Prometheus to observe and monitor streaming applications in an effective way.</dd> </dl> diff --git a/content/zh/index.html b/content/zh/index.html index 027ccec..5de9175 100644 --- a/content/zh/index.html +++ b/content/zh/index.html @@ -446,6 +446,9 @@ <dl> + <dt> <a href="/2019/05/19/state-ttl.html">State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</a></dt> + <dd>A common requirement for many stateful streaming applications is to automatically cleanup application state for effective management of your state size, or to control how long the application state can be accessed. State TTL enables application state cleanup and efficient state size management in Apache Flink</dd> + <dt> <a href="/2019/05/14/temporal-tables.html">Flux capacitor, huh? Temporal Tables and Joins in Streaming SQL</a></dt> <dd>Apache Flink natively supports temporal table joins since the 1.7 release for straightforward temporal data handling. In this blog post, we provide an overview of how this new concept can be leveraged for effective point-in-time analysis in streaming scenarios.</dd> @@ -465,9 +468,6 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa for more details.</p> </dd> - - <dt> <a href="/features/2019/03/11/prometheus-monitoring.html">Flink and Prometheus: Cloud-native monitoring of streaming applications</a></dt> - <dd>This blog post describes how developers can leverage Apache Flink's built-in metrics system together with Prometheus to observe and monitor streaming applications in an effective way.</dd> </dl>
