Rebuild website This closes #100.
Project: http://git-wip-us.apache.org/repos/asf/flink-web/repo Commit: http://git-wip-us.apache.org/repos/asf/flink-web/commit/035337bd Tree: http://git-wip-us.apache.org/repos/asf/flink-web/tree/035337bd Diff: http://git-wip-us.apache.org/repos/asf/flink-web/diff/035337bd Branch: refs/heads/asf-site Commit: 035337bd381b913ad2ec69aaa13f92061a1faff8 Parents: 8f98e45 Author: Fabian Hueske <[email protected]> Authored: Wed Feb 7 18:58:14 2018 +0100 Committer: Fabian Hueske <[email protected]> Committed: Wed Feb 7 18:58:30 2018 +0100 ---------------------------------------------------------------------- content/blog/feed.xml | 90 ++++++ content/blog/index.html | 44 ++- content/blog/page2/index.html | 46 +-- content/blog/page3/index.html | 45 ++- content/blog/page4/index.html | 44 ++- content/blog/page5/index.html | 46 +-- content/blog/page6/index.html | 32 ++- .../2018/01/30/incremental-checkpointing.html | 277 +++++++++++++++++++ .../img/blog/incremental_cp_impl_example.svg | 3 + content/index.html | 6 +- 10 files changed, 553 insertions(+), 80 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/flink-web/blob/035337bd/content/blog/feed.xml ---------------------------------------------------------------------- diff --git a/content/blog/feed.xml b/content/blog/feed.xml index 2006044..0c44c61 100644 --- a/content/blog/feed.xml +++ b/content/blog/feed.xml @@ -7,6 +7,96 @@ <atom:link href="http://flink.apache.org/blog/feed.xml"; rel="self" type="application/rss+xml" /> <item> +<title>Managing Large State in Apache Flink: An Intro to Incremental Checkpointing</title> +<description><p>Apache Flink was purpose-built for <em>stateful</em> stream processing. However, what is state in a stream processing application? I defined state and stateful stream processing in a <a href="http://flink.apache.org/features/2017/07/04/flink-rescalable-state.html">previous blog post</a>, and in case you need a refresher, <em>state is defined as memory in an applicationâs operators that stores information about previously-seen events that you can use to influence the processing of future events</em>.</p> + +<p>State is a fundamental, enabling concept in stream processing required for a majority of complex use cases. Some examples highlighted in the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/state.html">Flink documentation</a>:</p> + +<ul> + <li>When an application searches for certain event patterns, the state stores the sequence of events encountered so far.</li> + <li>When aggregating events per minute, the state holds the pending aggregates.</li> + <li>When training a machine learning model over a stream of data points, the state holds the current version of the model parameters.</li> +</ul> + +<p>However, stateful stream processing is only useful in production environments if the state is fault tolerant. âFault toleranceâ means that even if thereâs a software or machine failure, the computed end-result is accurate, with no data loss or double-counting of events.</p> + +<p>Flinkâs fault tolerance has always been a powerful and popular feature, minimizing the impact of software or machine failure on your business and making it possible to guarantee exactly-once results from a Flink application.</p> + +<p>Core to this is <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/checkpointing.html">checkpointing</a>, which is the mechanism Flink uses to make application state fault tolerant. A checkpoint in Flink is a global, asynchronous snapshot of application state thatâs taken on a regular interval and sent to durable storage (usually, a distributed file system). In the event of a failure, Flink restarts an application using the most recently completed checkpoint as a starting point. Some Apache Flink users run applications with gigabytes or even terabytes of application state. These users reported that with such large state, creating a checkpoint was often a slow and resource intensive operation, which is why in Flink 1.3 we introduced âincremental checkpointing.â</p> + +<p>Before incremental checkpointing, every single Flink checkpoint consisted of the full state of an application. We created the incremental checkpointing feature after we noticed that writing the full state for every checkpoint was often unnecessary, as the state changes from one checkpoint to the next were rarely that large. Incremental checkpointing instead maintains the differences (or âdeltaâ) between each checkpoint and stores only the differences between the last checkpoint and the current state.</p> + +<p>Incremental checkpoints can provide a significant performance improvement for jobs with a very large state. Early testing of the feature by a production user with terabytes of state shows a drop in checkpoint time from more than 3 minutes down to 30 seconds after implementing incremental checkpoints. This is because the checkpoint doesnât need to transfer the full state to durable storage on each checkpoint.</p> + +<h3 id="how-to-start">How to Start</h3> + +<p>Currently, you can only use incremental checkpointing with a RocksDB state back-end, and Flink uses RocksDBâs internal backup mechanism to consolidate checkpoint data over time. As a result, the incremental checkpoint history in Flink does not grow indefinitely, and Flink eventually consumes and prunes old checkpoints automatically.</p> + +<p>To enable incremental checkpointing in your application, I recommend you read the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/large_state_tuning.html#tuning-rocksdb">the Apache Flink documentation on checkpointing</a> for full details, but in summary, you enable checkpointing as normal, but enable incremental checkpointing in the constructor by setting the second parameter to <code>true</code>.</p> + +<h4 id="java-example">Java Example</h4> + +<div class="highlight"><pre><code class="language-java"><span class="n">StreamExecutionEnvironment</span> <span class="n">env</span> <span class="o">=</span> <span class="n">StreamExecutionEnvironment</span><span class="o">.</span><span class="na">getExecutionEnvironment</span><span class="o">();</span> +<span class="n">env</span><span class="o">.</span><span class="na">setStateBackend</span><span class="o">(</span><span class="k">new</span> <span class="nf">RocksDBStateBackend</span><span class="o">(</span><span class="n">filebackend</span><span class="o">,</span> <span class="kc">true</span><span class="o">));</span></code></pre></div> + +<h4 id="scala-example">Scala Example</h4> + +<div class="highlight"><pre><code class="language-scala"><span class="k">val</span> <span class="n">env</span> <span class="k">=</span> <span class="nc">StreamExecutionEnvironment</span><span class="o">.</span><span class="n">getExecutionEnvironment</span><span class="o">()</span> +<span class="n">env</span><span class="o">.</span><span class="n">setStateBackend</span><span class="o">(</span><span class="k">new</span> <span class="nc">RocksDBStateBackend</span><span class="o">(</span><span class="n">filebackend</span><span class="o">,</span> <span class="kc">true</span><span class="o">))</span></code></pre></div> + +<p>By default, Flink retains 1 completed checkpoint, so if you need a higher number, <a href="https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/state/checkpointing.html#related-config-options">you can configure it with the following flag</a>:</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">state</span><span class="o">.</span><span class="na">checkpoints</span><span class="o">.</span><span class="na">num</span><span class="o">-</span><span class="n">retained</span></code></pre></div> + +<h3 id="how-it-works">How it Works</h3> + +<p>Flinkâs incremental checkpointing uses <a href="https://github.com/facebook/rocksdb/wiki/Checkpoints">RocksDB checkpoints</a> as a foundation. RocksDB is a key-value store based on â<a href="https://en.wikipedia.org/wiki/Log-structured_merge-tree">log-structured-merge</a>â (LSM) trees that collects all changes in a mutable (changeable) in-memory buffer called a âmemtableâ. Any updates to the same key in the memtable replace previous values, and once the memtable is full, RocksDB writes it to disk with all entries sorted by their key and with light compression applied. Once RocksDB writes the memtable to disk it is immutable (unchangeable) and is now called a âsorted-string-tableâ (sstable).</p> + +<p>A âcompactionâ background task merges sstables to consolidate potential duplicates for each key, and over time RocksDB deletes the original sstables, with the merged sstable containing all information from across all the other sstables.</p> + +<p>On top of this, Flink tracks which sstable files RocksDB has created and deleted since the previous checkpoint, and as the sstables are immutable, Flink uses this to figure out the state changes. To do this, Flink triggers a flush in RocksDB, forcing all memtables into sstables on disk, and hard-linked in a local temporary directory. This process is synchronous to the processing pipeline, and Flink performs all further steps asynchronously and does not block processing.</p> + +<p>Then Flink copies all new sstables to stable storage (e.g., HDFS, S3) to reference in the new checkpoint. Flink doesnât copy all sstables that already existed in the previous checkpoint to stable storage but re-reference them. Any new checkpoints will no longer reference deleted files as deleted sstables in RocksDB are always the result of compaction, and it eventually replaces old tables with an sstable that is the result of a merge. This how in Flinkâs incremental checkpoints can prune the checkpoint history.</p> + +<p>For tracking changes between checkpoints, the uploading of consolidated tables is redundant work. Flink performs the process incrementally, and typically adds only a small overhead, so we consider this worthwhile because it allows Flink to keep a shorter history of checkpoints to consider in a recovery.</p> + +<h4 id="an-example">An Example</h4> + +<p><img src="/img/blog/incremental_cp_impl_example.svg" alt="Example setup" /> +<em>Example setup</em></p> + +<p>Take an example with a subtask of one operator that has a keyed state, and the number of retained checkpoints set at <strong>2</strong>. The columns in the figure above show the state of the local RocksDB instance for each checkpoint, the files it references, and the counts in the shared state registry after the checkpoint completes.</p> + +<p>For checkpoint âCP 1â, the local RocksDB directory contains two sstable files, it considers these new and uploads them to stable storage using directory names that match the checkpoint name. When the checkpoint completes, Flink creates the two entries in the shared state registry and sets their counts to â1â. The key in the shared state registry is a composite of an operator, subtask, and the original sstable file name. The registry also keeps a mapping from the key to the file path in stable storage.</p> + +<p>For checkpoint âCP 2â, RocksDB has created two new sstable files, and the two older ones still exist. For checkpoint âCP 2â, Flink adds the two new files to stable storage and can reference the previous two files. When the checkpoint completes, Flink increases the counts for all referenced files by 1.</p> + +<p>For checkpoint âCP 3â, RocksDBâs compaction has merged <code>sstable-(1)</code>, <code>sstable-(2)</code>, and <code>sstable-(3)</code> into <code>sstable-(1,2,3)</code> and deleted the original files. This merged file contains the same information as the source files, with all duplicate entries eliminated. In addition to this merged file, <code>sstable-(4)</code> still exists and there is now a new <code>sstable-(5)</code> file. Flink adds the new <code>sstable-(1,2,3)</code> and <code>sstable-(5)</code> files to stable storage, <code>sstable-(4)</code> is re-referenced from checkpoint âCP 2â and increases the counts for referenced files by 1. The older âCP 1â checkpoint is now deleted as the number of retained checkpoints (2) has been reached. As part of this deletion, Flink decreases the counts for all files referenced âCP 1â, (<code>sstable -(1)</code> and <code>sstable-(2)</code>), by 1.</p> + +<p>For checkpoint âCP-4â, RocksDB has merged <code>sstable-(4)</code>, <code>sstable-(5)</code>, and a new <code>sstable-(6)</code> into <code>sstable-(4,5,6)</code>. Flink adds this new table to stable storage and references it together with <code>sstable-(1,2,3)</code>, it increases the counts for <code>sstable-(1,2,3)</code> and <code>sstable-(4,5,6)</code> by 1 and then deletes âCP-2â as the number of retained checkpoints has been reached. As the counts for <code>sstable-(1)</code>, <code>sstable-(2)</code>, and <code>sstable-(3)</code> have now dropped to 0, and Flink deletes them from stable storage.</p> + +<h3 id="race-conditions-and-concurrent-checkpoints">Race Conditions and Concurrent Checkpoints</h3> + +<p>As Flink can execute multiple checkpoints in parallel, sometimes new checkpoints start before confirming previous checkpoints as completed. Because of this, you should consider which the previous checkpoint to use as a basis for a new incremental checkpoint. Flink only references state from a checkpoint confirmed by the checkpoint coordinator so that it doesnât unintentionally reference a deleted shared file.</p> + +<h3 id="restoring-checkpoints-and-performance-considerations">Restoring Checkpoints and Performance Considerations</h3> + +<p>If you enable incremental checkpointing, there are no further configuration steps needed to recover your state in case of failure. If a failure occurs, Flinkâs <code>JobManager</code> tells all tasks to restore from the last completed checkpoint, be it a full or incremental checkpoint. Each <code>TaskManager</code> then downloads their share of the state from the checkpoint on the distributed file system.</p> + +<p>Though the feature can lead to a substantial improvement in checkpoint time for users with a large state, there are trade-offs to consider with incremental checkpointing. Overall, the process reduces the checkpointing time during normal operations but can lead to a longer recovery time depending on the size of your state. If the cluster failure is particularly severe and the Flink <code>TaskManager</code>s have to read from multiple checkpoints, recovery can be a slower operation than when using non-incremental checkpointing. You can also no longer delete old checkpoints as newer checkpoints need them, and the history of differences between checkpoints can grow indefinitely over time. You need to plan for larger distributed storage to maintain the checkpoints and the network overhead to read from it.</p> + +<p>There are some strategies for improving the convenience/performance trade-off, and I recommend you read <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/checkpoints.html#basics-of-incremental-checkpoints">the Flink documentation</a> for more details.</p> + +<p><em>This post <a href="https://data-artisans.com/blog/managing-large-state-apache-flink-incremental-checkpointing-overview" target="_blank"> originally appeared on the data Artisans blog </a>and was contributed to the Flink blog by Stefan Richter and Chris Ward.</em></p> +<link rel="canonical" href="https://data-artisans.com/blog/managing-large-state-apache-flink-incremental-checkpointing-overview" /> + +</description> +<pubDate>Tue, 30 Jan 2018 13:00:00 +0100</pubDate> +<link>http://flink.apache.org/features/2018/01/30/incremental-checkpointing.html</link> +<guid isPermaLink="true">/features/2018/01/30/incremental-checkpointing.html</guid> +</item> + +<item> <title>Apache Flink in 2017: Year in Review</title> <description><p>2017 was another exciting year for the Apache Flink® community, with 3 major version releases (<a href="http://flink.apache.org/news/2017/02/06/release-1.2.0.html">Flink 1.2.0 in February</a>, <a href="http://flink.apache.org/news/2017/06/01/release-1.3.0.html">Flink 1.3.0 in June</a>, and <a href="http://flink.apache.org/news/2017/12/12/release-1.4.0.html">Flink 1.4.0 in December</a>) and the first-ever <a href="https://sf-2017.flink-forward.org/">Flink Forward in San Francisco</a>, giving Flink community members in another corner of the globe an opportunity to connect. Users shared details about their innovative production deployments, redefining what is possible with a modern stream processing framework like Flink.</p> http://git-wip-us.apache.org/repos/asf/flink-web/blob/035337bd/content/blog/index.html ---------------------------------------------------------------------- diff --git a/content/blog/index.html b/content/blog/index.html index 961d15e..6d7c615 100644 --- a/content/blog/index.html +++ b/content/blog/index.html @@ -142,6 +142,19 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/features/2018/01/30/incremental-checkpointing.html">Managing Large State in Apache Flink: An Intro to Incremental Checkpointing</a></h2> + + <p>30 Jan 2018 + Stefan Ricther (<a href="https://twitter.com/StefanRRicther";>@StefanRRicther</a>) & Chris Ward (<a href="https://twitter.com/chrischinch";>@chrischinch</a>)</p> + + <p>Flink 1.3.0 introduced incremental checkpointing, making it possible for applications with large state to generate checkpoints more efficiently.</p> + + <p><a href="/features/2018/01/30/incremental-checkpointing.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2017/12/21/2017-year-in-review.html">Apache Flink in 2017: Year in Review</a></h2> <p>21 Dec 2017 @@ -277,20 +290,6 @@ whatâs coming in Flink 1.4.0 as well as a preview of what the Flink community <hr> - <article> - <h2 class="blog-title"><a href="/news/2017/04/04/dynamic-tables.html">Continuous Queries on Dynamic Tables</a></h2> - - <p>04 Apr 2017 by Fabian Hueske, Shaoxuan Wang, and Xiaowei Jiang - </p> - - <p><p>Flink's relational APIs, the Table API and SQL, are unified APIs for stream and batch processing, meaning that a query produces the same result when being evaluated on streaming or static data.</p> -<p>In this blog post we discuss the future of these APIs and introduce the concept of Dynamic Tables. Dynamic tables will significantly expand the scope of the Table API and SQL on streams and enable many more advanced use cases. We discuss how streams and dynamic tables relate to each other and explain the semantics of continuously evaluating queries on dynamic tables.</p></p> - - <p><a href="/news/2017/04/04/dynamic-tables.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -319,10 +318,25 @@ whatâs coming in Flink 1.4.0 as well as a preview of what the Flink community - <h2>2017</h2> + <h2>2018</h2> <ul id="markdown-toc"> + <li><a href="/features/2018/01/30/incremental-checkpointing.html">Managing Large State in Apache Flink: An Intro to Incremental Checkpointing</a></li> + + + + </ul> + <hr> + <h2>2017</h2> + <ul id="markdown-toc"> + + + + + + + <li><a href="/news/2017/12/21/2017-year-in-review.html">Apache Flink in 2017: Year in Review</a></li> http://git-wip-us.apache.org/repos/asf/flink-web/blob/035337bd/content/blog/page2/index.html ---------------------------------------------------------------------- diff --git a/content/blog/page2/index.html b/content/blog/page2/index.html index 35840c1..bfac156 100644 --- a/content/blog/page2/index.html +++ b/content/blog/page2/index.html @@ -142,6 +142,20 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2017/04/04/dynamic-tables.html">Continuous Queries on Dynamic Tables</a></h2> + + <p>04 Apr 2017 by Fabian Hueske, Shaoxuan Wang, and Xiaowei Jiang + </p> + + <p><p>Flink's relational APIs, the Table API and SQL, are unified APIs for stream and batch processing, meaning that a query produces the same result when being evaluated on streaming or static data.</p> +<p>In this blog post we discuss the future of these APIs and introduce the concept of Dynamic Tables. Dynamic tables will significantly expand the scope of the Table API and SQL on streams and enable many more advanced use cases. We discuss how streams and dynamic tables relate to each other and explain the semantics of continuously evaluating queries on dynamic tables.</p></p> + + <p><a href="/news/2017/04/04/dynamic-tables.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2017/03/29/table-sql-api-update.html">From Streams to Tables and Back Again: An Update on Flink's Table & SQL API</a></h2> <p>29 Mar 2017 by Timo Walther (<a href="https://twitter.com/";>@twalthr</a>) @@ -270,21 +284,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2016/08/08/release-1.1.0.html">Announcing Apache Flink 1.1.0</a></h2> - - <p>08 Aug 2016 - </p> - - <p><div class="alert alert-success"><strong>Important</strong>: The Maven artifacts published with version 1.1.0 on Maven central have a Hadoop dependency issue. It is highly recommended to use <strong>1.1.1</strong> or <strong>1.1.1-hadoop1</strong> as the Flink version.</div> - -</p> - - <p><a href="/news/2016/08/08/release-1.1.0.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -313,9 +312,24 @@ - <h2>2017</h2> + <h2>2018</h2> + + <ul id="markdown-toc"> + + <li><a href="/features/2018/01/30/incremental-checkpointing.html">Managing Large State in Apache Flink: An Intro to Incremental Checkpointing</a></li> + + + </ul> + <hr> + <h2>2017</h2> <ul id="markdown-toc"> + + + + + + <li><a href="/news/2017/12/21/2017-year-in-review.html">Apache Flink in 2017: Year in Review</a></li> http://git-wip-us.apache.org/repos/asf/flink-web/blob/035337bd/content/blog/page3/index.html ---------------------------------------------------------------------- diff --git a/content/blog/page3/index.html b/content/blog/page3/index.html index 5e0a9e6..89bc68f 100644 --- a/content/blog/page3/index.html +++ b/content/blog/page3/index.html @@ -142,6 +142,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2016/08/08/release-1.1.0.html">Announcing Apache Flink 1.1.0</a></h2> + + <p>08 Aug 2016 + </p> + + <p><div class="alert alert-success"><strong>Important</strong>: The Maven artifacts published with version 1.1.0 on Maven central have a Hadoop dependency issue. It is highly recommended to use <strong>1.1.1</strong> or <strong>1.1.1-hadoop1</strong> as the Flink version.</div> + +</p> + + <p><a href="/news/2016/08/08/release-1.1.0.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2016/05/24/stream-sql.html">Stream Processing for Everyone with SQL and Apache Flink</a></h2> <p>24 May 2016 by Fabian Hueske (<a href="https://twitter.com/";>@fhueske</a>) @@ -271,19 +286,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2015/12/11/storm-compatibility.html">Storm Compatibility in Apache Flink: How to run existing Storm topologies on Flink</a></h2> - - <p>11 Dec 2015 by Matthias J. Sax (<a href="https://twitter.com/";>@MatthiasJSax</a>) - </p> - - <p>In this blog post, we describe Flink's compatibility package for <a href="https://storm.apache.org";>Apache Storm</a> that allows to embed Spouts (sources) and Bolts (operators) in a regular Flink streaming job. Furthermore, the compatibility package provides a Storm compatible API in order to execute whole Storm topologies with (almost) no code adaption.</p> - - <p><a href="/news/2015/12/11/storm-compatibility.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -312,9 +314,24 @@ - <h2>2017</h2> + <h2>2018</h2> + + <ul id="markdown-toc"> + + <li><a href="/features/2018/01/30/incremental-checkpointing.html">Managing Large State in Apache Flink: An Intro to Incremental Checkpointing</a></li> + + + </ul> + <hr> + <h2>2017</h2> <ul id="markdown-toc"> + + + + + + <li><a href="/news/2017/12/21/2017-year-in-review.html">Apache Flink in 2017: Year in Review</a></li> http://git-wip-us.apache.org/repos/asf/flink-web/blob/035337bd/content/blog/page4/index.html ---------------------------------------------------------------------- diff --git a/content/blog/page4/index.html b/content/blog/page4/index.html index 68ccf62..02cdc3a 100644 --- a/content/blog/page4/index.html +++ b/content/blog/page4/index.html @@ -142,6 +142,19 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2015/12/11/storm-compatibility.html">Storm Compatibility in Apache Flink: How to run existing Storm topologies on Flink</a></h2> + + <p>11 Dec 2015 by Matthias J. Sax (<a href="https://twitter.com/";>@MatthiasJSax</a>) + </p> + + <p>In this blog post, we describe Flink's compatibility package for <a href="https://storm.apache.org";>Apache Storm</a> that allows to embed Spouts (sources) and Bolts (operators) in a regular Flink streaming job. Furthermore, the compatibility package provides a Storm compatible API in order to execute whole Storm topologies with (almost) no code adaption.</p> + + <p><a href="/news/2015/12/11/storm-compatibility.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2015/12/04/Introducing-windows.html">Introducing Stream Windows in Apache Flink</a></h2> <p>04 Dec 2015 by Fabian Hueske (<a href="https://twitter.com/";>@fhueske</a>) @@ -279,20 +292,6 @@ vertex-centric or gather-sum-apply to Flink dataflows.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2015/05/11/Juggling-with-Bits-and-Bytes.html">Juggling with Bits and Bytes</a></h2> - - <p>11 May 2015 by Fabian Hüske (<a href="https://twitter.com/";>@fhueske</a>) - </p> - - <p><p>Nowadays, a lot of open-source systems for analyzing large data sets are implemented in Java or other JVM-based programming languages. The most well-known example is Apache Hadoop, but also newer frameworks such as Apache Spark, Apache Drill, and also Apache Flink run on JVMs. A common challenge that JVM-based data analysis engines face is to store large amounts of data in memory - both for caching and for efficient processing such as sorting and joining of data. Managing the JVM memory well makes the difference between a system that is hard to configure and has unpredictable reliability and performance and a system that behaves robustly with few configuration knobs.</p> -<p>In this blog post we discuss how Apache Flink manages memory, talk about its custom data de/serialization stack, and show how it operates on binary data.</p></p> - - <p><a href="/news/2015/05/11/Juggling-with-Bits-and-Bytes.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -321,10 +320,25 @@ vertex-centric or gather-sum-apply to Flink dataflows.</p> - <h2>2017</h2> + <h2>2018</h2> <ul id="markdown-toc"> + <li><a href="/features/2018/01/30/incremental-checkpointing.html">Managing Large State in Apache Flink: An Intro to Incremental Checkpointing</a></li> + + + + </ul> + <hr> + <h2>2017</h2> + <ul id="markdown-toc"> + + + + + + + <li><a href="/news/2017/12/21/2017-year-in-review.html">Apache Flink in 2017: Year in Review</a></li> http://git-wip-us.apache.org/repos/asf/flink-web/blob/035337bd/content/blog/page5/index.html ---------------------------------------------------------------------- diff --git a/content/blog/page5/index.html b/content/blog/page5/index.html index 12cdbe0..6828a8b 100644 --- a/content/blog/page5/index.html +++ b/content/blog/page5/index.html @@ -142,6 +142,20 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2015/05/11/Juggling-with-Bits-and-Bytes.html">Juggling with Bits and Bytes</a></h2> + + <p>11 May 2015 by Fabian Hüske (<a href="https://twitter.com/";>@fhueske</a>) + </p> + + <p><p>Nowadays, a lot of open-source systems for analyzing large data sets are implemented in Java or other JVM-based programming languages. The most well-known example is Apache Hadoop, but also newer frameworks such as Apache Spark, Apache Drill, and also Apache Flink run on JVMs. A common challenge that JVM-based data analysis engines face is to store large amounts of data in memory - both for caching and for efficient processing such as sorting and joining of data. Managing the JVM memory well makes the difference between a system that is hard to configure and has unpredictable reliability and performance and a system that behaves robustly with few configuration knobs.</p> +<p>In this blog post we discuss how Apache Flink manages memory, talk about its custom data de/serialization stack, and show how it operates on binary data.</p></p> + + <p><a href="/news/2015/05/11/Juggling-with-Bits-and-Bytes.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2015/04/13/release-0.9.0-milestone1.html">Announcing Flink 0.9.0-milestone1 preview release</a></h2> <p>13 Apr 2015 @@ -286,21 +300,6 @@ and offers a new API including definition of flexible windows.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2014/11/04/release-0.7.0.html">Apache Flink 0.7.0 available</a></h2> - - <p>04 Nov 2014 - </p> - - <p><p>We are pleased to announce the availability of Flink 0.7.0. This release includes new user-facing features as well as performance and bug fixes, brings the Scala and Java APIs in sync, and introduces Flink Streaming. A total of 34 people have contributed to this release, a big thanks to all of them!</p> - -</p> - - <p><a href="/news/2014/11/04/release-0.7.0.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -329,9 +328,24 @@ and offers a new API including definition of flexible windows.</p> - <h2>2017</h2> + <h2>2018</h2> + + <ul id="markdown-toc"> + + <li><a href="/features/2018/01/30/incremental-checkpointing.html">Managing Large State in Apache Flink: An Intro to Incremental Checkpointing</a></li> + + + </ul> + <hr> + <h2>2017</h2> <ul id="markdown-toc"> + + + + + + <li><a href="/news/2017/12/21/2017-year-in-review.html">Apache Flink in 2017: Year in Review</a></li> http://git-wip-us.apache.org/repos/asf/flink-web/blob/035337bd/content/blog/page6/index.html ---------------------------------------------------------------------- diff --git a/content/blog/page6/index.html b/content/blog/page6/index.html index c2cbd3f..c4522ea 100644 --- a/content/blog/page6/index.html +++ b/content/blog/page6/index.html @@ -142,6 +142,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2014/11/04/release-0.7.0.html">Apache Flink 0.7.0 available</a></h2> + + <p>04 Nov 2014 + </p> + + <p><p>We are pleased to announce the availability of Flink 0.7.0. This release includes new user-facing features as well as performance and bug fixes, brings the Scala and Java APIs in sync, and introduces Flink Streaming. A total of 34 people have contributed to this release, a big thanks to all of them!</p> + +</p> + + <p><a href="/news/2014/11/04/release-0.7.0.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2014/10/03/upcoming_events.html">Upcoming Events</a></h2> <p>03 Oct 2014 @@ -217,9 +232,24 @@ academic and open source project that Flink originates from.</p> - <h2>2017</h2> + <h2>2018</h2> + + <ul id="markdown-toc"> + + <li><a href="/features/2018/01/30/incremental-checkpointing.html">Managing Large State in Apache Flink: An Intro to Incremental Checkpointing</a></li> + + + </ul> + <hr> + <h2>2017</h2> <ul id="markdown-toc"> + + + + + + <li><a href="/news/2017/12/21/2017-year-in-review.html">Apache Flink in 2017: Year in Review</a></li> http://git-wip-us.apache.org/repos/asf/flink-web/blob/035337bd/content/features/2018/01/30/incremental-checkpointing.html ---------------------------------------------------------------------- diff --git a/content/features/2018/01/30/incremental-checkpointing.html b/content/features/2018/01/30/incremental-checkpointing.html new file mode 100644 index 0000000..1556d02 --- /dev/null +++ b/content/features/2018/01/30/incremental-checkpointing.html @@ -0,0 +1,277 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> + <title>Apache Flink: Managing Large State in Apache Flink: An Intro to Incremental Checkpointing</title> + <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> + <link rel="icon" href="/favicon.ico" type="image/x-icon"> + + <!-- Bootstrap --> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css";> + <link rel="stylesheet" href="/css/flink.css"> + <link rel="stylesheet" href="/css/syntax.css"> + + <!-- Blog RSS feed --> + <link href="/blog/feed.xml" rel="alternate" type="application/rss+xml" title="Apache Flink Blog: RSS feed" /> + + <!-- jQuery (necessary for Bootstrap's JavaScript plugins) --> + <!-- We need to load Jquery in the header for custom google analytics event tracking--> + <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.2/jquery.min.js";></script> + + <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> + <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> + <!--[if lt IE 9]> + <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js";></script> + <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js";></script> + <![endif]--> + </head> + <body> + + + <!-- Main content. --> + <div class="container"> + <div class="row"> + + + <div id="sidebar" class="col-sm-3"> + <!-- Top navbar. --> + <nav class="navbar navbar-default"> + <!-- The logo. --> + <div class="navbar-header"> + <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <div class="navbar-logo"> + <a href="/"> + <img alt="Apache Flink" src="/img/flink-header-logo.svg" width="147px" height="73px"> + </a> + </div> + </div><!-- /.navbar-header --> + + <!-- The navigation links. --> + <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1"> + <ul class="nav navbar-nav navbar-main"> + + <!-- Downloads --> + <li class=""><a class="btn btn-info" href="/downloads.html">Download Flink</a></li> + + <!-- Overview --> + <li><a href="/index.html">Home</a></li> + + <!-- Intro --> + <li><a href="/introduction.html">Introduction to Flink</a></li> + + <!-- Use cases --> + <li><a href="/usecases.html">Flink Use Cases</a></li> + + <!-- Powered by --> + <li><a href="/poweredby.html">Powered by Flink</a></li> + + <!-- Ecosystem --> + <li><a href="/ecosystem.html">Ecosystem</a></li> + + <!-- Community --> + <li><a href="/community.html">Community & Project Info</a></li> + + <!-- Contribute --> + <li><a href="/how-to-contribute.html">How to Contribute</a></li> + + <!-- Blog --> + <li class=" hidden-md hidden-sm"><a href="/blog/"><b>Flink Blog</b></a></li> + + <hr /> + + + + <!-- Documentation --> + <!-- <li> + <a href="http://ci.apache.org/projects/flink/flink-docs-release-1.4"; target="_blank">Documentation <small><span class="glyphicon glyphicon-new-window"></span></small></a> + </li> --> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation + <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="http://ci.apache.org/projects/flink/flink-docs-release-1.4"; target="_blank">1.4 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + <li><a href="http://ci.apache.org/projects/flink/flink-docs-master"; target="_blank">1.5 (Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + </ul> + </li> + + <!-- Quickstart --> + <li> + <a href="http://ci.apache.org/projects/flink/flink-docs-release-1.4/quickstart/setup_quickstart.html"; target="_blank">Quickstart <small><span class="glyphicon glyphicon-new-window"></span></small></a> + </li> + + <!-- GitHub --> + <li> + <a href="https://github.com/apache/flink"; target="_blank">Flink on GitHub <small><span class="glyphicon glyphicon-new-window"></span></small></a> + </li> + + </ul> + + + + <ul class="nav navbar-nav navbar-bottom"> + <hr /> + + <!-- FAQ --> + <li ><a href="/faq.html">Project FAQ</a></li> + + <!-- Twitter --> + <li><a href="https://twitter.com/apacheflink"; target="_blank">@ApacheFlink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + + <!-- Visualizer --> + <li class=" hidden-md hidden-sm"><a href="/visualizer/" target="_blank">Plan Visualizer <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + + </ul> + </div><!-- /.navbar-collapse --> + </nav> + + </div> + <div class="col-sm-9"> + <div class="row-fluid"> + <div class="col-sm-12"> + <div class="row"> + <h1>Managing Large State in Apache Flink: An Intro to Incremental Checkpointing</h1> + + <article> + <p>30 Jan 2018 Stefan Ricther (<a href="https://twitter.com/StefanRRicther";>@StefanRRicther</a>) & Chris Ward (<a href="https://twitter.com/chrischinch";>@chrischinch</a>)</p> + +<p>Apache Flink was purpose-built for <em>stateful</em> stream processing. However, what is state in a stream processing application? I defined state and stateful stream processing in a <a href="http://flink.apache.org/features/2017/07/04/flink-rescalable-state.html";>previous blog post</a>, and in case you need a refresher, <em>state is defined as memory in an applicationâs operators that stores information about previously-seen events that you can use to influence the processing of future events</em>.</p> + +<p>State is a fundamental, enabling concept in stream processing required for a majority of complex use cases. Some examples highlighted in the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/state.html";>Flink documentation</a>:</p> + +<ul> + <li>When an application searches for certain event patterns, the state stores the sequence of events encountered so far.</li> + <li>When aggregating events per minute, the state holds the pending aggregates.</li> + <li>When training a machine learning model over a stream of data points, the state holds the current version of the model parameters.</li> +</ul> + +<p>However, stateful stream processing is only useful in production environments if the state is fault tolerant. âFault toleranceâ means that even if thereâs a software or machine failure, the computed end-result is accurate, with no data loss or double-counting of events.</p> + +<p>Flinkâs fault tolerance has always been a powerful and popular feature, minimizing the impact of software or machine failure on your business and making it possible to guarantee exactly-once results from a Flink application.</p> + +<p>Core to this is <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/checkpointing.html";>checkpointing</a>, which is the mechanism Flink uses to make application state fault tolerant. A checkpoint in Flink is a global, asynchronous snapshot of application state thatâs taken on a regular interval and sent to durable storage (usually, a distributed file system). In the event of a failure, Flink restarts an application using the most recently completed checkpoint as a starting point. Some Apache Flink users run applications with gigabytes or even terabytes of application state. These users reported that with such large state, creating a checkpoint was often a slow and resource intensive operation, which is why in Flink 1.3 we introduced âincremental checkpointing.â</p> + +<p>Before incremental checkpointing, every single Flink checkpoint consisted of the full state of an application. We created the incremental checkpointing feature after we noticed that writing the full state for every checkpoint was often unnecessary, as the state changes from one checkpoint to the next were rarely that large. Incremental checkpointing instead maintains the differences (or âdeltaâ) between each checkpoint and stores only the differences between the last checkpoint and the current state.</p> + +<p>Incremental checkpoints can provide a significant performance improvement for jobs with a very large state. Early testing of the feature by a production user with terabytes of state shows a drop in checkpoint time from more than 3 minutes down to 30 seconds after implementing incremental checkpoints. This is because the checkpoint doesnât need to transfer the full state to durable storage on each checkpoint.</p> + +<h3 id="how-to-start">How to Start</h3> + +<p>Currently, you can only use incremental checkpointing with a RocksDB state back-end, and Flink uses RocksDBâs internal backup mechanism to consolidate checkpoint data over time. As a result, the incremental checkpoint history in Flink does not grow indefinitely, and Flink eventually consumes and prunes old checkpoints automatically.</p> + +<p>To enable incremental checkpointing in your application, I recommend you read the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/large_state_tuning.html#tuning-rocksdb";>the Apache Flink documentation on checkpointing</a> for full details, but in summary, you enable checkpointing as normal, but enable incremental checkpointing in the constructor by setting the second parameter to <code>true</code>.</p> + +<h4 id="java-example">Java Example</h4> + +<div class="highlight"><pre><code class="language-java"><span class="n">StreamExecutionEnvironment</span> <span class="n">env</span> <span class="o">=</span> <span class="n">StreamExecutionEnvironment</span><span class="o">.</span><span class="na">getExecutionEnvironment</span><span class="o">();</span> +<span class="n">env</span><span class="o">.</span><span class="na">setStateBackend</span><span class="o">(</span><span class="k">new</span> <span class="nf">RocksDBStateBackend</span><span class="o">(</span><span class="n">filebackend</span><span class="o">,</span> <span class="kc">true</span><span class="o">));</span></code></pre></div> + +<h4 id="scala-example">Scala Example</h4> + +<div class="highlight"><pre><code class="language-scala"><span class="k">val</span> <span class="n">env</span> <span class="k">=</span> <span class="nc">StreamExecutionEnvironment</span><span class="o">.</span><span class="n">getExecutionEnvironment</span><span class="o">()</span> +<span class="n">env</span><span class="o">.</span><span class="n">setStateBackend</span><span class="o">(</span><span class="k">new</span> <span class="nc">RocksDBStateBackend</span><span class="o">(</span><span class="n">filebackend</span><span class="o">,</span> <span class="kc">true</span><span class="o">))</span></code></pre></div> + +<p>By default, Flink retains 1 completed checkpoint, so if you need a higher number, <a href="https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/state/checkpointing.html#related-config-options";>you can configure it with the following flag</a>:</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">state</span><span class="o">.</span><span class="na">checkpoints</span><span class="o">.</span><span class="na">num</span><span class="o">-</span><span class="n">retained</span></code></pre></div> + +<h3 id="how-it-works">How it Works</h3> + +<p>Flinkâs incremental checkpointing uses <a href="https://github.com/facebook/rocksdb/wiki/Checkpoints";>RocksDB checkpoints</a> as a foundation. RocksDB is a key-value store based on â<a href="https://en.wikipedia.org/wiki/Log-structured_merge-tree";>log-structured-merge</a>â (LSM) trees that collects all changes in a mutable (changeable) in-memory buffer called a âmemtableâ. Any updates to the same key in the memtable replace previous values, and once the memtable is full, RocksDB writes it to disk with all entries sorted by their key and with light compression applied. Once RocksDB writes the memtable to disk it is immutable (unchangeable) and is now called a âsorted-string-tableâ (sstable).</p> + +<p>A âcompactionâ background task merges sstables to consolidate potential duplicates for each key, and over time RocksDB deletes the original sstables, with the merged sstable containing all information from across all the other sstables.</p> + +<p>On top of this, Flink tracks which sstable files RocksDB has created and deleted since the previous checkpoint, and as the sstables are immutable, Flink uses this to figure out the state changes. To do this, Flink triggers a flush in RocksDB, forcing all memtables into sstables on disk, and hard-linked in a local temporary directory. This process is synchronous to the processing pipeline, and Flink performs all further steps asynchronously and does not block processing.</p> + +<p>Then Flink copies all new sstables to stable storage (e.g., HDFS, S3) to reference in the new checkpoint. Flink doesnât copy all sstables that already existed in the previous checkpoint to stable storage but re-reference them. Any new checkpoints will no longer reference deleted files as deleted sstables in RocksDB are always the result of compaction, and it eventually replaces old tables with an sstable that is the result of a merge. This how in Flinkâs incremental checkpoints can prune the checkpoint history.</p> + +<p>For tracking changes between checkpoints, the uploading of consolidated tables is redundant work. Flink performs the process incrementally, and typically adds only a small overhead, so we consider this worthwhile because it allows Flink to keep a shorter history of checkpoints to consider in a recovery.</p> + +<h4 id="an-example">An Example</h4> + +<p><img src="/img/blog/incremental_cp_impl_example.svg" alt="Example setup" /> +<em>Example setup</em></p> + +<p>Take an example with a subtask of one operator that has a keyed state, and the number of retained checkpoints set at <strong>2</strong>. The columns in the figure above show the state of the local RocksDB instance for each checkpoint, the files it references, and the counts in the shared state registry after the checkpoint completes.</p> + +<p>For checkpoint âCP 1â, the local RocksDB directory contains two sstable files, it considers these new and uploads them to stable storage using directory names that match the checkpoint name. When the checkpoint completes, Flink creates the two entries in the shared state registry and sets their counts to â1â. The key in the shared state registry is a composite of an operator, subtask, and the original sstable file name. The registry also keeps a mapping from the key to the file path in stable storage.</p> + +<p>For checkpoint âCP 2â, RocksDB has created two new sstable files, and the two older ones still exist. For checkpoint âCP 2â, Flink adds the two new files to stable storage and can reference the previous two files. When the checkpoint completes, Flink increases the counts for all referenced files by 1.</p> + +<p>For checkpoint âCP 3â, RocksDBâs compaction has merged <code>sstable-(1)</code>, <code>sstable-(2)</code>, and <code>sstable-(3)</code> into <code>sstable-(1,2,3)</code> and deleted the original files. This merged file contains the same information as the source files, with all duplicate entries eliminated. In addition to this merged file, <code>sstable-(4)</code> still exists and there is now a new <code>sstable-(5)</code> file. Flink adds the new <code>sstable-(1,2,3)</code> and <code>sstable-(5)</code> files to stable storage, <code>sstable-(4)</code> is re-referenced from checkpoint âCP 2â and increases the counts for referenced files by 1. The older âCP 1â checkpoint is now deleted as the number of retained checkpoints (2) has been reached. As part of this deletion, Flink decreases the counts for all files referenced âCP 1â, (<code>sstable-(1)</code> and <code>sstable-(2)</code>), by 1.</p> + +<p>For checkpoint âCP-4â, RocksDB has merged <code>sstable-(4)</code>, <code>sstable-(5)</code>, and a new <code>sstable-(6)</code> into <code>sstable-(4,5,6)</code>. Flink adds this new table to stable storage and references it together with <code>sstable-(1,2,3)</code>, it increases the counts for <code>sstable-(1,2,3)</code> and <code>sstable-(4,5,6)</code> by 1 and then deletes âCP-2â as the number of retained checkpoints has been reached. As the counts for <code>sstable-(1)</code>, <code>sstable-(2)</code>, and <code>sstable-(3)</code> have now dropped to 0, and Flink deletes them from stable storage.</p> + +<h3 id="race-conditions-and-concurrent-checkpoints">Race Conditions and Concurrent Checkpoints</h3> + +<p>As Flink can execute multiple checkpoints in parallel, sometimes new checkpoints start before confirming previous checkpoints as completed. Because of this, you should consider which the previous checkpoint to use as a basis for a new incremental checkpoint. Flink only references state from a checkpoint confirmed by the checkpoint coordinator so that it doesnât unintentionally reference a deleted shared file.</p> + +<h3 id="restoring-checkpoints-and-performance-considerations">Restoring Checkpoints and Performance Considerations</h3> + +<p>If you enable incremental checkpointing, there are no further configuration steps needed to recover your state in case of failure. If a failure occurs, Flinkâs <code>JobManager</code> tells all tasks to restore from the last completed checkpoint, be it a full or incremental checkpoint. Each <code>TaskManager</code> then downloads their share of the state from the checkpoint on the distributed file system.</p> + +<p>Though the feature can lead to a substantial improvement in checkpoint time for users with a large state, there are trade-offs to consider with incremental checkpointing. Overall, the process reduces the checkpointing time during normal operations but can lead to a longer recovery time depending on the size of your state. If the cluster failure is particularly severe and the Flink <code>TaskManager</code>s have to read from multiple checkpoints, recovery can be a slower operation than when using non-incremental checkpointing. You can also no longer delete old checkpoints as newer checkpoints need them, and the history of differences between checkpoints can grow indefinitely over time. You need to plan for larger distributed storage to maintain the checkpoints and the network overhead to read from it.</p> + +<p>There are some strategies for improving the convenience/performance trade-off, and I recommend you read <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/checkpoints.html#basics-of-incremental-checkpoints";>the Flink documentation</a> for more details.</p> + +<p><em>This post <a href="https://data-artisans.com/blog/managing-large-state-apache-flink-incremental-checkpointing-overview"; target="_blank"> originally appeared on the data Artisans blog </a>and was contributed to the Flink blog by Stefan Richter and Chris Ward.</em></p> +<link rel="canonical" href="https://data-artisans.com/blog/managing-large-state-apache-flink-incremental-checkpointing-overview"; /> + + + </article> + </div> + + <div class="row"> + <div id="disqus_thread"></div> + <script type="text/javascript"> + /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */ + var disqus_shortname = 'stratosphere-eu'; // required: replace example with your forum shortname + + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); + </script> + </div> + </div> +</div> + </div> + </div> + + <hr /> + + <div class="row"> + <div class="footer text-center col-sm-12"> + <p>Copyright © 2014-2017 <a href="http://apache.org";>The Apache Software Foundation</a>. All Rights Reserved.</p> + <p>Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.</p> + <p><a href="/privacy-policy.html">Privacy Policy</a> · <a href="/blog/feed.xml">RSS feed</a></p> + </div> + </div> + </div><!-- /.container --> + + <!-- Include all compiled plugins (below), or include individual files as needed --> + <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js";></script> + <script src="/js/codetabs.js"></script> + <script src="/js/stickysidebar.js"></script> + + + <!-- Google Analytics --> + <script> + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + + ga('create', 'UA-52545728-1', 'auto'); + ga('send', 'pageview'); + </script> + </body> +</html> http://git-wip-us.apache.org/repos/asf/flink-web/blob/035337bd/content/img/blog/incremental_cp_impl_example.svg ---------------------------------------------------------------------- diff --git a/content/img/blog/incremental_cp_impl_example.svg b/content/img/blog/incremental_cp_impl_example.svg new file mode 100644 index 0000000..5c852ab --- /dev/null +++ b/content/img/blog/incremental_cp_impl_example.svg @@ -0,0 +1,3 @@ +<?xml version="1.0"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd";> +<svg xmlns="http://www.w3.org/2000/svg"; xmlns:xl="http://www.w3.org/1999/xlink"; version="1.1" viewBox="5.456693 3543.4567 789.0866 317.08661" width="789.0866pt" height="317.08661pt"><metadata xmlns:dc="http://purl.org/dc/elements/1.1/";><dc:date>2017-08-08 10:07Z</dc:date><!-- Produced by OmniGraffle Professional 5.4.4 --></metadata><defs><font-face font-family="Monaco" font-size="7" units-per-em="1000" underline-position="-37.597656" underline-thickness="75.683594" slope="0" x-height="560.54688" cap-height="780.27344" ascent="1e3" descent="-250" font-weight="500"><font-face-src><font-face-name name="Monaco"/></font-face-src></font-face><font-face font-family="Helvetica" font-size="12" units-per-em="1000" underline-position="-75.683594" underline-thickness="49.316406" slope="0" x-height="532.22656" cap-height="719.72656" ascent="770.01953" descent="-229.98047" font-weight="bold"><font-face-src><font-face-name name="Helvetica-Bold"/></font-face-src></font-face><font-face font-family=" Helvetica" font-size="9" units-per-em="1000" underline-position="-75.683594" underline-thickness="49.316406" slope="0" x-height="522.94922" cap-height="717.28516" ascent="770.01953" descent="-229.98047" font-weight="500"><font-face-src><font-face-name name="Helvetica"/></font-face-src></font-face><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="DimensionArrow_Marker" viewBox="-1 -5 7 10" markerWidth="7" markerHeight="10" color="#738a05"><g><path d="M 0 0 L 4.8000002 0 M 4.8000002 3.0000001 L 4.8000002 -3.0000001 M 0 1.20000005 L 4.2000002 0 L 0 -1.20000005" fill="none" stroke="currentColor" stroke-width="1"/></g></marker><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="DimensionArrow_Marker_2" viewBox="-6 -5 7 10" markerWidth="7" markerHeight="10" color="#738a05"><g><path d="M 0 0 L -4.8000002 0 M -4.8000002 -3.0000001 L -4.8000002 3.0000001 M 0 -1.20000005 L -4.2000002 0 L 0 1.20000005" fill="none" stroke="currentColor" stroke-width="1"/> </g></marker><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="DimensionArrow_Marker_3" viewBox="-1 -5 7 10" markerWidth="7" markerHeight="10" color="#bd3612"><g><path d="M 0 0 L 4.8000002 0 M 4.8000002 3.0000001 L 4.8000002 -3.0000001 M 0 1.20000005 L 4.2000002 0 L 0 -1.20000005" fill="none" stroke="currentColor" stroke-width="1"/></g></marker><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="DimensionArrow_Marker_4" viewBox="-6 -5 7 10" markerWidth="7" markerHeight="10" color="#bd3612"><g><path d="M 0 0 L -4.8000002 0 M -4.8000002 -3.0000001 L -4.8000002 3.0000001 M 0 -1.20000005 L -4.2000002 0 L 0 1.20000005" fill="none" stroke="currentColor" stroke-width="1"/></g></marker><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="DimensionArrow_Marker_5" viewBox="-1 -5 7 10" markerWidth="7" markerHeight="10" color="#2076c8"><g><path d="M 0 0 L 4.8000002 0 M 4.8000002 3.0000001 L 4.8000002 -3.0000001 M 0 1.20000005 L 4.2000002 0 L 0 -1.20000005" fill="none" stroke="currentColor" stroke-width="1"/></g></marker><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="DimensionArrow_Marker_6" viewBox="-6 -5 7 10" markerWidth="7" markerHeight="10" color="#2076c8"><g><path d="M 0 0 L -4.8000002 0 M -4.8000002 -3.0000001 L -4.8000002 3.0000001 M 0 -1.20000005 L -4.2000002 0 L 0 1.20000005" fill="none" stroke="currentColor" stroke-width="1"/></g></marker><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="DimensionArrow_Marker_7" viewBox="-1 -5 7 10" markerWidth="7" markerHeight="10" color="#a57706"><g><path d="M 0 0 L 4.8000002 0 M 4.8000002 3.0000001 L 4.8000002 -3.0000001 M 0 1.20000005 L 4.2000002 0 L 0 -1.20000005" fill="none" stroke="currentColor" stroke-width="1"/></g></marker><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="DimensionArrow_Marker_8" viewBox="-6 -5 7 10" markerWidth="7" markerHeight="10" color="#a57706"><g><path d="M 0 0 L -4.8000002 0 M -4.8000002 -3.0000001 L -4.8000002 3.0000001 M 0 -1.20000005 L -4.2000002 0 L 0 1.20000005" fill="none" stroke="currentColor" stroke-width="1"/></g></marker></defs><g stroke="none" stroke-opacity="1" stroke-dasharray="none" fill="none" fill-opacity="1"><title>Arbeitsfläche 1</title><rect fill="white" width="1118.55115" height="3914.4489"/><g><title>Ebene 1</title><line x1="232.43229" y1="3744.6237" x2="165.44272" y2="3815.9998" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><line x1="160.75" y1="3744.624" x2="160.75" y2="3815.9998" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><line x1="84.375015" y1="3663.7481" x2="84.375015" y2="3734.6237" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><line x1="156.02824" y1="3663.7482" x2="89.09676" y2="3734.6237" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><line x1="227.68149" y1="3663.7481" x2="93.81851" y2="3734.6237" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="49" y="3729.8736" width="70.75" height="18.500217" fill="#0a2832"/><rect x="49" y="3729.8736" width="70.75" height="18.500217" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(49 3734.8315)" fill="white"><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="3.869873" y="7" textLength="63.010254">sstable-(1,2,3)</tspan></text><rect x="49" y="3589.749" width="70.75" height="18.500217" fill="#0a2832"/><rect x="49" y="3589.749" width="70.75" height="18.500217" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(49 3594.707)" fill="white"><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="12.27124" y="7" textLength="46.20752">sstable-(1)</tspan></text><rect x="125.875" y="3589.749" width="70.75" height="18.500217" fill="#0a2832" /><rect x="125.875" y="3589.749" width="70.75" height="18.500217" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(125.875 3594.707)" fill="white"><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="12.27124" y="7" textLength="46.20752">sstable-(2)</tspan></text><rect x="49" y="3648.998" width="70.75" height="18.500217" fill="#0a2832"/><rect x="49" y="3648.998" width="70.75" height="18.500217" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(49 3653.9559)" fill="white"><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="12.27124" y="7" textLength="46.20752">sstable-(1)</tspan></text><rect x="125.375" y="3648.998" width="70.75" height="18.500217" fill="#0a2832"/><rect x="125.375" y="3648.998" width="70.75" height="18.500217" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="tr anslate(125.375 3653.956)" fill="white"><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="12.27124" y="7" textLength="46.20752">sstable-(2)</tspan></text><rect x="201.75" y="3648.998" width="70.75" height="18.500217" fill="#0a2832"/><rect x="201.75" y="3648.998" width="70.75" height="18.500217" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(201.75 3653.9559)" fill="white"><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="12.27124" y="7" textLength="46.20752">sstable-(3)</tspan></text><rect x="278.125" y="3648.998" width="70.75" height="18.500217" fill="#0a2832"/><rect x="278.125" y="3648.998" width="70.75" height="18.500217" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(278.125 3653.956)" fill="white"><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="12.27124" y="7" textLength="46.20752" >sstable-(4)</tspan></text><rect x="125.375" y="3729.8737" width="70.75" >height="18.500217" fill="#0a2832"/><rect x="125.375" y="3729.8737" >width="70.75" height="18.500217" stroke="black" stroke-linecap="round" >stroke-linejoin="round" stroke-width="1"/><text transform="translate(125.375 >3734.8317)" fill="white"><tspan font-family="Monaco" font-size="7" >font-weight="500" fill="white" x="12.27124" y="7" >textLength="46.20752">sstable-(4)</tspan></text><rect x="201.75" >y="3729.8735" width="70.75" height="18.500217" fill="#0a2832"/><rect >x="201.75" y="3729.8735" width="70.75" height="18.500217" stroke="black" >stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text >transform="translate(201.75 3734.8314)" fill="white"><tspan >font-family="Monaco" font-size="7" font-weight="500" fill="white" >x="12.27124" y="7" textLength="46.20752">sstable-(5)</tspan></text><text >transform="translate(11.5 3591.999)" fill="#738a05"><tspan >font-family="Helvetica" font-size="12" font-weight="bold " fill="#738a05" x=".26660156" y="11" textLength="16.669922">CP</tspan><tspan font-family="Helvetica" font-size="12" font-weight="bold" fill="#738a05" x="16.725586" y="11" textLength="10.0078125"> 1</tspan></text><text transform="translate(11 3651.248)" fill="#bd3612"><tspan font-family="Helvetica" font-size="12" font-weight="bold" fill="#bd3612" x=".26660156" y="11" textLength="16.669922">CP</tspan><tspan font-family="Helvetica" font-size="12" font-weight="bold" fill="#bd3612" x="16.725586" y="11" textLength="10.0078125"> 2</tspan></text><text transform="translate(11 3732.1238)" fill="#2076c8"><tspan font-family="Helvetica" font-size="12" font-weight="bold" fill="#2076c8" x=".26660156" y="11" textLength="16.669922">CP</tspan><tspan font-family="Helvetica" font-size="12" font-weight="bold" fill="#2076c8" x="16.725586" y="11" textLength="10.0078125"> 3</tspan></text><text transform="translate(89.5 3671.686)" fill="black"><tspan font-family="Helvetica" font-size="9" font-weight="500" x=".24487305" y="9" textLength="25.510254">merge</tspan></text><rect x="360.65693" y="3581.999" width="207.84307" height="34.000217" fill="black"/><rect x="360.65693" y="3581.999" width="207.84307" height="34.000217" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(365.65693 3587.207)" fill="#738a05"><tspan font-family="Monaco" font-size="7" font-weight="500" fill="#738a05" x="0" y="7" textLength="172.22803">op-2-1-sstable-(1) -> dfs://cp-1/23-23-42</tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="#738a05" x="0" y="21" textLength="29.404785">op-2-1-</tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="#738a05" x="29.404785" y="21" textLength="63.010254">sstable-(2) -> </tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="#738a05" x="92.41504" y="21" textLength="79.81299">dfs://cp-1/54-75-91</tspan></text><rect x="360.65693" y="3625.998" width="207.84308" he ight="65.75202" fill="black"/><rect x="360.65693" y="3625.998" width="207.84308" height="65.75202" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(365.65693 3633.0818)" fill="#738a05"><tspan font-family="Monaco" font-size="7" font-weight="500" fill="#738a05" x="0" y="7" textLength="180.62939">op-2-1-sstable-(1) -> (dfs://cp-1/23-23-42)</tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="#738a05" x="0" y="21" textLength="180.62939">op-2-1-sstable-(2) -> (dfs://cp-1/54-75-91)</tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="#bd3612" x="0" y="35" textLength="172.22803">op-2-1-sstable-(3) -> dfs://cp-2/44-63-90</tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="#bd3612" x="0" y="49" textLength="172.22803">op-2-1-sstable-(4) -> dfs://cp-2/41-74-28</tspan></text><rect x="360.65693" y="3702.2488" width="207.84308" height="56.5" fill="black"/><rect x="360 .65693" y="3702.2488" width="207.84308" height="56.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(365.65693 3711.7065)" fill="#2076c8"><tspan font-family="Monaco" font-size="7" font-weight="500" fill="#2076c8" x="0" y="7" textLength="189.03076">op-2-1-sstable-(1,2,3) -> dfs://cp-3/38-46-83</tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="#bd3612" x="0" y="21" textLength="180.62939">op-2-1-sstable-(4) -> (dfs://cp-2/41-74-28)</tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="#2076c8" x="0" y="35" textLength="29.404785">op-2-1-</tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="#2076c8" x="29.404785" y="35" textLength="155.42529">sstable-(5) -> dfs://cp-3/79-20-41-55</tspan></text><rect x="577.81385" y="3581.999" width="123.62231" height="34.000286" fill="black"/><rect x="577.81385" y="3581.999" width="123.62231" height="34.000286" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(582.81385 3587.2069)" fill="white"><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="0" y="7" textLength="92.41504">op-2-1-sstable-(1) : 1</tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="0" y="21" textLength="92.41504">op-2-1-sstable-(2) : 1</tspan></text><rect x="579.28193" y="3625.998" width="122.15423" height="65.75202" fill="black"/><rect x="579.28193" y="3625.998" width="122.15423" height="65.75202" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(584.28193 3633.0818)" fill="white"><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="0" y="7" textLength="92.41504">op-2-1-sstable-(1) : 2</tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="0" y="21" textLength="92.41504">op-2-1-sstable-(2) : 2</tspan><tspan font-family ="Monaco" font-size="7" font-weight="500" fill="white" x="0" y="35" textLength="92.41504">op-2-1-sstable-(3) : 1</tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="0" y="49" textLength="92.41504">op-2-1-sstable-(4) : 1</tspan></text><rect x="579.28193" y="3702.2488" width="122.15423" height="82.25122" fill="black"/><rect x="579.28193" y="3702.2488" width="122.15423" height="82.25122" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(584.28193 3703.5822)" fill="white"><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="0" y="7" textLength="92.41504">op-2-1-sstable-(1) : 1</tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="0" y="21" textLength="92.41504">op-2-1-sstable-(2) : 1</tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="0" y="35" textLength="92.41504">op-2-1-sstable-(3) : 1</tspan><tspan font-family="Mon aco" font-size="7" font-weight="500" fill="white" x="0" y="49" textLength="109.217773">op-2-1-sstable-(1,2,3) : 1</tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="0" y="63" textLength="92.41504">op-2-1-sstable-(4) : 2</tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="0" y="77" textLength="92.41504">op-2-1-sstable-(5) : 1</tspan></text><rect x="49" y="3811.2495" width="70.75" height="18.500217" fill="#0a2832"/><rect x="49" y="3811.2495" width="70.75" height="18.500217" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(49 3816.2074)" fill="white"><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="3.869873" y="7" textLength="63.010254">sstable-(1,2,3)</tspan></text><rect x="125.375" y="3811.2496" width="70.75" height="18.500217" fill="#0a2832"/><rect x="125.375" y="3811.2496" width="70.75" height="18.500217" stroke="black" stroke-linecap=" round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(125.375 3816.2076)" fill="white"><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="3.869873" y="7" textLength="63.010254">sstable-(4,5,6)</tspan></text><text transform="translate(11.5 3813.4997)" fill="#a57706"><tspan font-family="Helvetica" font-size="12" font-weight="bold" fill="#a57706" x=".26660156" y="11" textLength="16.669922">CP</tspan><tspan font-family="Helvetica" font-size="12" font-weight="bold" fill="#a57706" x="16.725586" y="11" textLength="10.0078125"> 4</tspan></text><text transform="translate(166 3752.5619)" fill="black"><tspan font-family="Helvetica" font-size="9" font-weight="500" x=".24487305" y="9" textLength="25.510254">merge</tspan></text><rect x="360.65693" y="3794" width="207.84308" height="33.5001" fill="black"/><rect x="360.65693" y="3794" width="207.84308" height="33.5001" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><tex t transform="translate(365.65693 3798.9578)" fill="#2076c8"><tspan font-family="Monaco" font-size="7" font-weight="500" fill="#2076c8" x="0" y="7" textLength="197.43213">op-2-1-sstable-(1,2,3) -> (dfs://cp-3/38-46-83)</tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="#a57706" x="0" y="21" textLength="189.03076">op-2-1-sstable-(4,5,6) -> dfs://cp-3/38-46-83</tspan></text><rect x="579.28193" y="3794" width="122.15423" height="54.743706" fill="black"/><rect x="579.28193" y="3794" width="122.15423" height="54.743706" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(584.28193 3795.5796)" fill="white"><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="0" y="7" textLength="109.217773">op-2-1-sstable-(1,2,3) : 2</tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="0" y="21" textLength="92.41504">op-2-1-sstable-(4) : 1</tspan><tspan font-family="Monaco " font-size="7" font-weight="500" fill="white" x="0" y="35" textLength="92.41504">op-2-1-sstable-(5) : 1</tspan><tspan font-family="Monaco" font-size="7" font-weight="500" fill="white" x="0" y="49" textLength="109.217773">op-2-1-sstable-(4,5,6) : 1</tspan></text><line x1="11" y1="3620.75" x2="788" y2="3620.75" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1" stroke-dasharray="1,4"/><line x1="11" y1="3697.25" x2="788" y2="3697.25" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1" stroke-dasharray="1,4"/><line x1="11" y1="3788.75" x2="788" y2="3788.75" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1" stroke-dasharray="1,4"/><line x1="717.2181" y1="3589.599" x2="717.2181" y2="3609.6928" marker-end="url(#DimensionArrow_Marker)" marker-start="url(#DimensionArrow_Marker_2)" stroke="#738a05" stroke-linecap="round" stroke-linejoin="round" stroke-width="2"/><line x1="738.25" y1="3589.5993" x2="738.25" y2 ="3685.65" marker-end="url(#DimensionArrow_Marker_3)" marker-start="url(#DimensionArrow_Marker_4)" stroke="#bd3612" stroke-linecap="round" stroke-linejoin="round" stroke-width="2"/><line x1="759.16273" y1="3631.8496" x2="757.83727" y2="3777.6504" marker-end="url(#DimensionArrow_Marker_5)" marker-start="url(#DimensionArrow_Marker_6)" stroke="#2076c8" stroke-linecap="round" stroke-linejoin="round" stroke-width="2"/><line x1="779.5" y1="3708.35" x2="779.5" y2="3840.1437" marker-end="url(#DimensionArrow_Marker_7)" marker-start="url(#DimensionArrow_Marker_8)" stroke="#a57706" stroke-linecap="round" stroke-linejoin="round" stroke-width="2"/><text transform="translate(121 3547)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x="2.1445312" y="11" textLength="146.04492">Local RocksDB directory </tspan><tspan font-family="Helvetica" font-size="12" font-weight="bold" x="1.8134766" y="25" textLength="143.37305">for Operator 2, Subtask 1</tspan></text><text transfo rm="translate(383.07846 3547.0001)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".1484375" y="11" textLength="162.703125">Shared States in Checkpoint</tspan><tspan font-family="Helvetica" font-size="12" font-weight="bold" x="6.9921875" y="25" textLength="135.01172">(Key -> uploaded DFS fi</tspan><tspan font-family="Helvetica" font-size="12" font-weight="bold" x="142.00391" y="25" textLength="14.003906">le)</tspan></text><text transform="translate(593.125 3547.0003)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x="9.8203125" y="11" textLength="76.69336">Shared State </tspan><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16113281" y="25" textLength="92.677734">Registry Counts</tspan></text><text transform="translate(712 3547.0003)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x="11.161133" y="11" textLength="54.01172">Retained </tspan><tspan font-family="Helve tica" font-size="12" font-weight="bold" x=".49414062" y="25" textLength="72.01172">Checkpoints</tspan></text><line x1="42" y1="3550.5" x2="42" y2="3850.7437" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1" stroke-dasharray="1,4"/><line x1="354.5" y1="3550.5" x2="354.5" y2="3849.7437" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1" stroke-dasharray="1,4"/><line x1="574" y1="3550.5" x2="574" y2="3848.7437" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1" stroke-dasharray="1,4"/><line x1="706.875" y1="3550.5" x2="706.875" y2="3848.7437" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1" stroke-dasharray="1,4"/><line x1="12" y1="3578.2088" x2="789" y2="3578.2088" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1" stroke-dasharray="1,4"/><text transform="translate(191 3796.4586)" fill="black"><tspan font-family="Monaco" font-size="7" font-wei ght="500" fill="black" x=".29589844" y="7" textLength="50.408203">+sstable-(6)</tspan></text></g></g></svg> http://git-wip-us.apache.org/repos/asf/flink-web/blob/035337bd/content/index.html ---------------------------------------------------------------------- diff --git a/content/index.html b/content/index.html index ce6ac23..6bce04e 100644 --- a/content/index.html +++ b/content/index.html @@ -168,6 +168,9 @@ <dl> + <dt> <a href="/features/2018/01/30/incremental-checkpointing.html">Managing Large State in Apache Flink: An Intro to Incremental Checkpointing</a></dt> + <dd>Flink 1.3.0 introduced incremental checkpointing, making it possible for applications with large state to generate checkpoints more efficiently.</dd> + <dt> <a href="/news/2017/12/21/2017-year-in-review.html">Apache Flink in 2017: Year in Review</a></dt> <dd>As 2017 comes to a close, let's take a moment to look back on the Flink community's great work during the past year.</dd> @@ -190,9 +193,6 @@ whatâs coming in Flink 1.4.0 as well as a preview of what the Flink community <dd><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.3 series.</p> </dd> - - <dt> <a href="/features/2017/07/04/flink-rescalable-state.html">A Deep Dive into Rescalable State in Apache Flink</a></dt> - <dd><p>A primer on stateful stream processing and an in-depth walkthrough of rescalable state in Apache Flink.</p></dd> </dl>
