http://git-wip-us.apache.org/repos/asf/flink-web/blob/9ec0a879/content/blog/feed.xml ---------------------------------------------------------------------- diff --git a/content/blog/feed.xml b/content/blog/feed.xml new file mode 100644 index 0000000..066ae2f --- /dev/null +++ b/content/blog/feed.xml @@ -0,0 +1,6041 @@ +<?xml version="1.0" encoding="UTF-8"?> +<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom";> +<channel> +<title>Flink Blog Feed</title> +<description>Flink Blog</description> +<link>http://flink.apache.org/blog</link> +<atom:link href="http://flink.apache.org/blog/feed.xml"; rel="self" type="application/rss+xml" /> + +<item> +<title>Apache Flink 1.1.4 Released</title> +<description><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.1 series.</p> + +<p>This release includes major robustness improvements for checkpoint cleanup on failures and consumption of intermediate streams. We highly recommend all users to upgrade to Flink 1.1.4.</p> + +<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span> + <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span> + <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span> + <span class="nt">&lt;version&gt;</span>1.1.4<span class="nt">&lt;/version&gt;</span> +<span class="nt">&lt;/dependency&gt;</span> +<span class="nt">&lt;dependency&gt;</span> + <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span> + <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.10<span class="nt">&lt;/artifactId&gt;</span> + <span class="nt">&lt;version&gt;</span>1.1.4<span class="nt">&lt;/version&gt;</span> +<span class="nt">&lt;/dependency&gt;</span> +<span class="nt">&lt;dependency&gt;</span> + <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span> + <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.10<span class="nt">&lt;/artifactId&gt;</span> + <span class="nt">&lt;version&gt;</span>1.1.4<span class="nt">&lt;/version&gt;</span> +<span class="nt">&lt;/dependency&gt;</span></code></pre></div> + +<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p> + +<h2 id="note-for-rocksdb-backend-users">Note for RocksDB Backend Users</h2> + +<p>We updated Flinkâs RocksDB dependency version from <code>4.5.1</code> to <code>4.11.2</code>. Between these versions some of RocksDBâs internal configuration defaults changed that would affect the memory footprint of running Flink with RocksDB. Therefore, we manually reset them to the previous defaults. If you want to run with the new Rocks 4.11.2 defaults, you can do this via:</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">RocksDBStateBackend</span> <span class="n">backend</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">RocksDBStateBackend</span><span class="o">(</span><span class="s">&quot;...&quot;</span><span class="o">);</span> +<span class="c1">// Use the new default options. Otherwise, the default for RocksDB 4.5.1</span> +<span class="c1">// `PredefinedOptions.DEFAULT_ROCKS_4_5_1` will be used.</span> +<span class="n">backend</span><span class="o">.</span><span class="na">setPredefinedOptions</span><span class="o">(</span><span class="n">PredefinedOptions</span><span class="o">.</span><span class="na">DEFAULT</span><span class="o">);</span></code></pre></div> + +<h2 id="release-notes---flink---version-114">Release Notes - Flink - Version 1.1.4</h2> + +<h3 id="sub-task">Sub-task</h3> +<ul> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4510">FLINK-4510</a>] - Always create CheckpointCoordinator +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4984">FLINK-4984</a>] - Add Cancellation Barriers to BarrierTracker and BarrierBuffer +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4985">FLINK-4985</a>] - Report Declined/Canceled Checkpoints to Checkpoint Coordinator +</li> +</ul> + +<h3 id="bug">Bug</h3> +<ul> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2662">FLINK-2662</a>] - CompilerException: &quot;Bug: Plan generation for Unions picked a ship strategy between binary plan operators.&quot; +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3680">FLINK-3680</a>] - Remove or improve (not set) text in the Job Plan UI +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3813">FLINK-3813</a>] - YARNSessionFIFOITCase.testDetachedMode failed on Travis +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4108">FLINK-4108</a>] - NPE in Row.productArity +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4506">FLINK-4506</a>] - CsvOutputFormat defaults allowNullValues to false, even though doc and declaration says true +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4581">FLINK-4581</a>] - Table API throws &quot;No suitable driver found for jdbc:calcite&quot; +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4586">FLINK-4586</a>] - NumberSequenceIterator and Accumulator threading issue +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4619">FLINK-4619</a>] - JobManager does not answer to client when restore from savepoint fails +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4727">FLINK-4727</a>] - Kafka 0.9 Consumer should also checkpoint auto retrieved offsets even when no data is read +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4862">FLINK-4862</a>] - NPE on EventTimeSessionWindows with ContinuousEventTimeTrigger +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4932">FLINK-4932</a>] - Don&#39;t let ExecutionGraph fail when in state Restarting +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4933">FLINK-4933</a>] - ExecutionGraph.scheduleOrUpdateConsumers can fail the ExecutionGraph +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4977">FLINK-4977</a>] - Enum serialization does not work in all cases +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4991">FLINK-4991</a>] - TestTask hangs in testWatchDogInterruptsTask +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4998">FLINK-4998</a>] - ResourceManager fails when num task slots &gt; Yarn vcores +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5013">FLINK-5013</a>] - Flink Kinesis connector doesn&#39;t work on old EMR versions +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5028">FLINK-5028</a>] - Stream Tasks must not go through clean shutdown logic on cancellation +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5038">FLINK-5038</a>] - Errors in the &quot;cancelTask&quot; method prevent closeables from being closed early +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5039">FLINK-5039</a>] - Avro GenericRecord support is broken +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5040">FLINK-5040</a>] - Set correct input channel types with eager scheduling +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5050">FLINK-5050</a>] - JSON.org license is CatX +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5057">FLINK-5057</a>] - Cancellation timeouts are picked from wrong config +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5058">FLINK-5058</a>] - taskManagerMemory attribute set wrong value in FlinkShell +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5063">FLINK-5063</a>] - State handles are not properly cleaned up for declined or expired checkpoints +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5073">FLINK-5073</a>] - ZooKeeperCompleteCheckpointStore executes blocking delete operation in ZooKeeper client thread +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5075">FLINK-5075</a>] - Kinesis consumer incorrectly determines shards as newly discovered when tested against Kinesalite +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5082">FLINK-5082</a>] - Pull ExecutionService lifecycle management out of the JobManager +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5085">FLINK-5085</a>] - Execute CheckpointCoodinator&#39;s state discard calls asynchronously +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5114">FLINK-5114</a>] - PartitionState update with finished execution fails +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5142">FLINK-5142</a>] - Resource leak in CheckpointCoordinator +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5149">FLINK-5149</a>] - ContinuousEventTimeTrigger doesn&#39;t fire at the end of the window +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5154">FLINK-5154</a>] - Duplicate TypeSerializer when writing RocksDB Snapshot +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5158">FLINK-5158</a>] - Handle ZooKeeperCompletedCheckpointStore exceptions in CheckpointCoordinator +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5172">FLINK-5172</a>] - In RocksDBStateBackend, set flink-core and flink-streaming-java to &quot;provided&quot; +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5173">FLINK-5173</a>] - Upgrade RocksDB dependency +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5184">FLINK-5184</a>] - Error result of compareSerialized in RowComparator class +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5193">FLINK-5193</a>] - Recovering all jobs fails completely if a single recovery fails +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5197">FLINK-5197</a>] - Late JobStatusChanged messages can interfere with running jobs +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5214">FLINK-5214</a>] - Clean up checkpoint files when failing checkpoint operation on TM +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5215">FLINK-5215</a>] - Close checkpoint streams upon cancellation +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5216">FLINK-5216</a>] - CheckpointCoordinator&#39;s &#39;minPauseBetweenCheckpoints&#39; refers to checkpoint start rather then checkpoint completion +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5218">FLINK-5218</a>] - Eagerly close checkpoint streams on cancellation +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5228">FLINK-5228</a>] - LocalInputChannel re-trigger request and release deadlock +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5229">FLINK-5229</a>] - Cleanup StreamTaskStates if a checkpoint operation of a subsequent operator fails +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5246">FLINK-5246</a>] - Don&#39;t discard unknown checkpoint messages in the CheckpointCoordinator +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5248">FLINK-5248</a>] - SavepointITCase doesn&#39;t catch savepoint restore failure +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5274">FLINK-5274</a>] - LocalInputChannel throws NPE if partition reader is released +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5275">FLINK-5275</a>] - InputChanelDeploymentDescriptors throws misleading Exception if producer failed/cancelled +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5276">FLINK-5276</a>] - ExecutionVertex archiving can throw NPE with many previous attempts +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5285">FLINK-5285</a>] - CancelCheckpointMarker flood when using at least once mode +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5326">FLINK-5326</a>] - IllegalStateException: Bug in Netty consumer logic: reader queue got notified by partition about available data, but none was available +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5352">FLINK-5352</a>] - Restore RocksDB 1.1.3 memory behavior +</li> +</ul> + +<h3 id="improvement">Improvement</h3> +<ul> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3347">FLINK-3347</a>] - TaskManager (or its ActorSystem) need to restart in case they notice quarantine +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3787">FLINK-3787</a>] - Yarn client does not report unfulfillable container constraints +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4445">FLINK-4445</a>] - Ignore unmatched state when restoring from savepoint +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4715">FLINK-4715</a>] - TaskManager should commit suicide after cancellation failure +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4894">FLINK-4894</a>] - Don&#39;t block on buffer request after broadcastEvent +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4975">FLINK-4975</a>] - Add a limit for how much data may be buffered during checkpoint alignment +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4996">FLINK-4996</a>] - Make CrossHint @Public +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5046">FLINK-5046</a>] - Avoid redundant serialization when creating the TaskDeploymentDescriptor +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5123">FLINK-5123</a>] - Add description how to do proper shading to Flink docs. +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5169">FLINK-5169</a>] - Make consumption of input channels fair +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5192">FLINK-5192</a>] - Provide better log config templates +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5194">FLINK-5194</a>] - Log heartbeats on TRACE level +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5196">FLINK-5196</a>] - Don&#39;t log InputChannelDescriptor +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5198">FLINK-5198</a>] - Overwrite TaskState toString +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5199">FLINK-5199</a>] - Improve logging of submitted job graph actions in HA case +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5201">FLINK-5201</a>] - Promote loaded config properties to INFO +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5207">FLINK-5207</a>] - Decrease HadoopFileSystem logging +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5249">FLINK-5249</a>] - description of datastream rescaling doesn&#39;t match the figure +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5259">FLINK-5259</a>] - wrong execution environment in retry delays example +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5278">FLINK-5278</a>] - Improve Task and checkpoint logging +</li> +</ul> + +<h3 id="new-feature">New Feature</h3> +<ul> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4976">FLINK-4976</a>] - Add a way to abort in flight checkpoints +</li> +</ul> + +<h3 id="task">Task</h3> +<ul> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4778">FLINK-4778</a>] - Update program example in /docs/setup/cli.md due to the change in FLINK-2021 +</li> +</ul> + +</description> +<pubDate>Wed, 21 Dec 2016 10:00:00 +0100</pubDate> +<link>http://flink.apache.org/news/2016/12/21/release-1.1.4.html</link> +<guid isPermaLink="true">/news/2016/12/21/release-1.1.4.html</guid> +</item> + +<item> +<title>Apache Flink in 2016: Year in Review</title> +<description><p>2016 was an exciting year for the Apache Flink® community, and the + <a href="http://flink.apache.org/news/2016/03/08/release-1.0.0.html" target="_blank">release of Flink 1.0 in March</a> + marked the first time in Flinkâs history that the community guaranteed API backward compatibility for all + versions in a series. This step forward for Flink was followed by many new and exciting production deployments + in organizations of all shapes and sizes, all around the globe.</p> + +<p>In this post, weâll look back on the projectâs progress over the course of 2016, and +weâll also preview what 2017 has in store.</p> + +<div class="page-toc"> +<ul id="markdown-toc"> + <li><a href="#community-growth" id="markdown-toc-community-growth">Community Growth</a> <ul> + <li><a href="#github" id="markdown-toc-github">Github</a></li> + <li><a href="#meetups" id="markdown-toc-meetups">Meetups</a></li> + </ul> + </li> + <li><a href="#flink-forward-2016" id="markdown-toc-flink-forward-2016">Flink Forward 2016</a></li> + <li><a href="#features-and-ecosystem" id="markdown-toc-features-and-ecosystem">Features and Ecosystem</a> <ul> + <li><a href="#flink-ecosystem-growth" id="markdown-toc-flink-ecosystem-growth">Flink Ecosystem Growth</a></li> + <li><a href="#feature-timeline-in-2016" id="markdown-toc-feature-timeline-in-2016">Feature Timeline in 2016</a></li> + </ul> + </li> + <li><a href="#looking-ahead-to-2017" id="markdown-toc-looking-ahead-to-2017">Looking ahead to 2017</a></li> +</ul> + +</div> + +<h2 id="community-growth">Community Growth</h2> + +<h3 id="github">Github</h3> +<p>First, hereâs a summary of community statistics from <a href="https://github.com/apache/flink" target="_blank">GitHub</a>. At the time of writing:</p> +<ul> + <li><b>Contributors</b> have increased from 150 in December 2015 to 258 in December 2016 (up <b>72%</b>)</li> + <li><b>Stars</b> have increased from 813 in December 2015 to 1830 in December 2016 (up <b>125%</b>)</li> + <li><b>Forks</b> have increased from 544 in December 2015 to 1255 in December 2016 (up <b>130%</b>)</li> +</ul> + +<p>The community also welcomed <b>3 new committers in 2016</b>: Chengxiang Li, Greg Hogan, and Tzu-Li (Gordon) Tai.</p> + +<p><br /><img src="/img/blog/github-stats-2016.png" width="775" alt="Apache Flink GitHub Stats" /> +<br /> +<br /></p> + +<p>Next, letâs take a look at a few other project stats, starting with number of commits. If we run:</p> + +<div class="highlight"><pre><code>git log --pretty=oneline --after=12/31/2015 | wc -l +</code></pre></div> +<p>â¦inside the Flink repository, weâll see a total of <strong>1884</strong> commits so far in 2016, bringing the all-time total commits to <strong>10,015</strong>.</p> + +<p>Now, letâs go a bit deeper. And here are instructions in case youâd like to take a look at this data yourself.</p> + +<ul> + <li>Download gitstats from the <a href="http://gitstats.sourceforge.net/">project homepage</a>. Or, on OS X with homebrew, type:</li> +</ul> + +<div class="highlight"><pre><code>brew install --HEAD homebrew/head-only/gitstats +</code></pre></div> + +<ul> + <li>Clone the Apache Flink git repository:</li> +</ul> + +<div class="highlight"><pre><code>git clone g...@github.com:apache/flink.git +</code></pre></div> + +<ul> + <li>Generate the statistics</li> +</ul> + +<div class="highlight"><pre><code>gitstats flink/ flink-stats/ +</code></pre></div> + +<ul> + <li>View all the statistics as an html page using your defaulf browser:</li> +</ul> + +<div class="highlight"><pre><code>open flink-stats/index.html +</code></pre></div> +<p>2016 is the year that Flink surpassed 1 million lines of code, now clocking in at <strong>1,034,137</strong> lines.</p> + +<p><img src="/img/blog/flink-lines-of-code-2016.png" align="center" width="550" alt="Flink Total Lines of Code" /></p> + +<p>Monday remains the day of the week with the most commits over the projectâs history:</p> + +<p><img src="/img/blog/flink-dow-2016.png" align="center" width="550" alt="Flink Commits by Day of Week" /></p> + +<p>And 5pm is still solidly the preferred commit time:</p> + +<p><img src="/img/blog/flink-hod-2016.png" align="center" width="550" alt="Flink Commits by Hour of Day" /></p> + +<p><br /></p> + +<h3 id="meetups">Meetups</h3> +<p><a href="https://www.meetup.com/topics/apache-flink/" target="_blank">Apache Flink Meetup membership</a> grew by <b>240%</b> +this year, and at the time of writing, there are 41 meetups comprised of 16,541 members listing Flink as a topicâup from 16 groups with 4,864 members in December 2015. +The Flink community is proud to be truly global in nature.</p> + +<p><img src="/img/blog/flink-meetups-dec2016.png" width="775" alt="Apache Flink Meetup Map" /></p> + +<h2 id="flink-forward-2016">Flink Forward 2016</h2> + +<p>The <a href="http://2016.flink-forward.org/" target="_blank">second annual Flink Forward conference </a>took place in +Berlin on September 12-14, and over 350 members of the Flink community came together for speaker sessions, training, +and discussion about Flink. <a href="http://2016.flink-forward.org/program/sessions/" target="_blank">Slides and videos</a> + from speaker sessions are available online, and we encourage you to take a look if youâre interested in learning more + about how Flink is used in production in a wide range of organizations.</p> + +<p>Flink Forward will be expanding to <a href="http://sf.flink-forward.org/" target="_blank">San Francisco in April 2017</a>, and the <a href="http://berlin.flink-forward.org/" target="_blank">third-annual Berlin event + is scheduled for September 2017.</a></p> + +<p><img src="/img/blog/speaker-logos-ff2016.png" width="775" alt="Flink Forward Speakers" /></p> + +<h2 id="features-and-ecosystem">Features and Ecosystem</h2> + +<h3 id="flink-ecosystem-growth">Flink Ecosystem Growth</h3> + +<p>Flink was added to a selection of distributions during 2016, making it easier +for an even larger base of users to start working with Flink:</p> + +<ul> + <li><a href="https://aws.amazon.com/blogs/big-data/use-apache-flink-on-amazon-emr/" target="_blank"> + Amazon EMR</a></li> + <li><a href="https://cloud.google.com/dataproc/docs/release-notes/service#november_29_2016" target="_blank"> + Google Cloud Dataproc</a></li> + <li><a href="https://www.lightbend.com/blog/introducing-lightbend-fast-data-platform" target="_blank"> + Lightbend Fast Data Platform</a></li> +</ul> + +<p>In addition, the Apache Beam and Flink communities teamed up to build a Flink runner for Beam that, according to the Google team, is <a href="https://cloud.google.com/blog/big-data/2016/05/why-apache-beam-a-google-perspective" target="_blank">âsophisticated enough to be a compelling alternative to Cloud Dataflow when running on premise or on non-Google cloudsâ</a>.</p> + +<h3 id="feature-timeline-in-2016">Feature Timeline in 2016</h3> + +<p>Hereâs a selection of major features added to Flink over the course of 2016:</p> + +<p><img src="/img/blog/flink-releases-2016.png" width="775" alt="Flink Release Timeline 2016" /></p> + +<p>If you spend time in the <a href="https://issues.apache.org/jira/browse/FLINK-4554?jql=project%20%3D%20FLINK%20AND%20issuetype%20%3D%20%22New%20Feature%22%20AND%20status%20%3D%20Resolved%20ORDER%20BY%20resolved%20DESC" target="_blank">Apache Flink JIRA project</a>, youâll see that the Flink community has addressed every single one of the roadmap items identified +in <a href="http://flink.apache.org/news/2015/12/18/a-year-in-review.html" target="_blank">2015âs year in review post</a>. Hereâs to making that an annual tradition. :)</p> + +<h2 id="looking-ahead-to-2017">Looking ahead to 2017</h2> + +<p>A good source of information about the Flink communityâs roadmap is the list of +<a href="https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals" target="_blank">Flink +Improvement Proposals (FLIPs)</a> in the project wiki. Below, weâll highlight a selection of FLIPs +that have been accepted by the community as well as some that are still under discussion.</p> + +<p>We should note that work is already underway on a number of these features, and some will even be included in Flink 1.2 at the beginning of 2017.</p> + +<ul> + <li> + <p><strong>A new Flink deployment and process model</strong>, as described in <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077" target="_blank">FLIP-6<a></a>. This work ensures that Flink supports a wide +range of deployment types and cluster managers, making it possible to run Flink smoothly in any environment.</a></p> + </li> + <li> + <p><strong>Dynamic scaling</strong> for both key-value state <a href="https://github.com/apache/flink/pull/2440" target="_blank">(as described in +this PR)<a></a> <em>and</em> non-partitioned state <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-8%3A+Rescalable+Non-Partitioned+State" target="_blank">(as described in FLIP-8)<a></a>, ensuring that itâs always possible to split or merge state when scaling up or down, respectively.</a></a></p> + </li> + <li> + <p><strong>Asynchronous I/O</strong>, as described in <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673" target="_blank">FLIP-12 +</a>, which makes I/O access a less time-consuming process without adding complexity or the need for extra checkpoint coordination.</p> + </li> + <li> + <p><strong>Enhancements to the window evictor</strong>, as described in <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-4+%3A+Enhance+Window+Evictor" target="_blank">FLIP-4</a>, +to provide users with more control over how elements are evicted from a window.</p> + </li> + <li> + <p><strong>Fined-grained recovery from task failures</strong>, as described in <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-1+%3A+Fine+Grained+Recovery+from+Task+Failures" target="_blank">FLIP-1</a>, +to make it possible to restart only what needs to be restarted during recovery, building on cached intermediate results.</p> + </li> + <li> + <p><strong>Unified checkpoints and savepoints</strong>, as described in <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-10%3A+Unify+Checkpoints+and+Savepoints" target="_blank">FLIP-10</a>, to +allow savepoints to be triggered automaticallyâimportant for program updates for the sake of error handling because savepoints allow the user to modify both + the job and Flink version whereas checkpoints can only be recovered with the same job.</p> + </li> + <li> + <p><strong>Table API window aggregations</strong>, as described in <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-11%3A+Table+API+Stream+Aggregations" target="_blank">FLIP-11</a>, to support group-window and row-window aggregates on streaming and batch tables.</p> + </li> + <li> + <p><strong>Side inputs</strong>, as described in <a href="https://docs.google.com/document/d/1hIgxi2Zchww_5fWUHLoYiXwSBXjv-M5eOv-MKQYN3m4/edit" target="_blank">this design document</a>, to +enable the joining of a main, high-throughput stream with one more more inputs with static or slowly-changing data.</p> + </li> +</ul> + +<p>If youâre interested in getting involved with Flink, we encourage you to take a look at the FLIPs and to join the discussion via the <a href="http://flink.apache.org/community.html#mailing-lists">Flink mailing lists</a>.</p> + +<p>Lastly, weâd like to extend a sincere thank you to all of the Flink community for making 2016 a great year!</p> +</description> +<pubDate>Mon, 19 Dec 2016 10:00:00 +0100</pubDate> +<link>http://flink.apache.org/news/2016/12/19/2016-year-in-review.html</link> +<guid isPermaLink="true">/news/2016/12/19/2016-year-in-review.html</guid> +</item> + +<item> +<title>Apache Flink 1.1.3 Released</title> +<description><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.1. series.</p> + +<p>We recommend all users to upgrade to Flink 1.1.3.</p> + +<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span> + <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span> + <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span> + <span class="nt">&lt;version&gt;</span>1.1.3<span class="nt">&lt;/version&gt;</span> +<span class="nt">&lt;/dependency&gt;</span> +<span class="nt">&lt;dependency&gt;</span> + <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span> + <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.10<span class="nt">&lt;/artifactId&gt;</span> + <span class="nt">&lt;version&gt;</span>1.1.3<span class="nt">&lt;/version&gt;</span> +<span class="nt">&lt;/dependency&gt;</span> +<span class="nt">&lt;dependency&gt;</span> + <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span> + <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.10<span class="nt">&lt;/artifactId&gt;</span> + <span class="nt">&lt;version&gt;</span>1.1.3<span class="nt">&lt;/version&gt;</span> +<span class="nt">&lt;/dependency&gt;</span></code></pre></div> + +<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p> + +<h2 id="note-for-rocksdb-backend-users">Note for RocksDB Backend Users</h2> + +<p>It is highly recommended to use the âfully asyncâ mode for the RocksDB state backend. The âfully asyncâ mode will most likely allow you to easily upgrade to Flink 1.2 (via <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/savepoints.html">savepoints</a>) when it is released. The âsemi asyncâ mode will no longer be supported by Flink 1.2.</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">RocksDBStateBackend</span> <span class="n">backend</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">RocksDBStateBackend</span><span class="o">(</span><span class="s">&quot;...&quot;</span><span class="o">);</span> +<span class="n">backend</span><span class="o">.</span><span class="na">enableFullyAsyncSnapshots</span><span class="o">();</span></code></pre></div> + +<h2 id="release-notes---flink---version-113">Release Notes - Flink - Version 1.1.3</h2> + +<h2> Bug +</h2> +<ul> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2662">FLINK-2662</a>] - CompilerException: &quot;Bug: Plan generation for Unions picked a ship strategy between binary plan operators.&quot; +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4311">FLINK-4311</a>] - TableInputFormat fails when reused on next split +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4329">FLINK-4329</a>] - Fix Streaming File Source Timestamps/Watermarks Handling +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4485">FLINK-4485</a>] - Finished jobs in yarn session fill /tmp filesystem +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4513">FLINK-4513</a>] - Kafka connector documentation refers to Flink 1.1-SNAPSHOT +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4514">FLINK-4514</a>] - ExpiredIteratorException in Kinesis Consumer on long catch-ups to head of stream +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4540">FLINK-4540</a>] - Detached job execution may prevent cluster shutdown +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4544">FLINK-4544</a>] - TaskManager metrics are vulnerable to custom JMX bean installation +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4566">FLINK-4566</a>] - ProducerFailedException does not properly preserve Exception causes +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4588">FLINK-4588</a>] - Fix Merging of Covering Window in MergingWindowSet +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4589">FLINK-4589</a>] - Fix Merging of Covering Window in MergingWindowSet +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4616">FLINK-4616</a>] - Kafka consumer doesn&#39;t store last emmited watermarks per partition in state +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4618">FLINK-4618</a>] - FlinkKafkaConsumer09 should start from the next record on startup from offsets in Kafka +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4619">FLINK-4619</a>] - JobManager does not answer to client when restore from savepoint fails +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4636">FLINK-4636</a>] - AbstractCEPPatternOperator fails to restore state +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4640">FLINK-4640</a>] - Serialization of the initialValue of a Fold on WindowedStream fails +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4651">FLINK-4651</a>] - Re-register processing time timers at the WindowOperator upon recovery. +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4663">FLINK-4663</a>] - Flink JDBCOutputFormat logs wrong WARN message +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4672">FLINK-4672</a>] - TaskManager accidentally decorates Kill messages +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4677">FLINK-4677</a>] - Jars with no job executions produces NullPointerException in ClusterClient +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4702">FLINK-4702</a>] - Kafka consumer must commit offsets asynchronously +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4727">FLINK-4727</a>] - Kafka 0.9 Consumer should also checkpoint auto retrieved offsets even when no data is read +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4732">FLINK-4732</a>] - Maven junction plugin security threat +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4777">FLINK-4777</a>] - ContinuousFileMonitoringFunction may throw IOException when files are moved +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4788">FLINK-4788</a>] - State backend class cannot be loaded, because fully qualified name converted to lower-case +</li> +</ul> + +<h2> Improvement +</h2> +<ul> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4396">FLINK-4396</a>] - GraphiteReporter class not found at startup of jobmanager +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4574">FLINK-4574</a>] - Strengthen fetch interval implementation in Kinesis consumer +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4723">FLINK-4723</a>] - Unify behaviour of committed offsets to Kafka / ZK for Kafka 0.8 and 0.9 consumer +</li> +</ul> +</description> +<pubDate>Wed, 12 Oct 2016 11:00:00 +0200</pubDate> +<link>http://flink.apache.org/news/2016/10/12/release-1.1.3.html</link> +<guid isPermaLink="true">/news/2016/10/12/release-1.1.3.html</guid> +</item> + +<item> +<title>Apache Flink 1.1.2 Released</title> +<description><p>The Apache Flink community released another bugfix version of the Apache Flink 1.1. series.</p> + +<p>We recommend all users to upgrade to Flink 1.1.2.</p> + +<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span> + <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span> + <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span> + <span class="nt">&lt;version&gt;</span>1.1.2<span class="nt">&lt;/version&gt;</span> +<span class="nt">&lt;/dependency&gt;</span> +<span class="nt">&lt;dependency&gt;</span> + <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span> + <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.10<span class="nt">&lt;/artifactId&gt;</span> + <span class="nt">&lt;version&gt;</span>1.1.2<span class="nt">&lt;/version&gt;</span> +<span class="nt">&lt;/dependency&gt;</span> +<span class="nt">&lt;dependency&gt;</span> + <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span> + <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.10<span class="nt">&lt;/artifactId&gt;</span> + <span class="nt">&lt;version&gt;</span>1.1.2<span class="nt">&lt;/version&gt;</span> +<span class="nt">&lt;/dependency&gt;</span></code></pre></div> + +<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p> + +<h2>Release Notes - Flink - Version 1.1.2</h2> + +<ul> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4236">FLINK-4236</a>] - Flink Dashboard stops showing list of uploaded jars if main method cannot be looked up +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4309">FLINK-4309</a>] - Potential null pointer dereference in DelegatingConfiguration#keySet() +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4334">FLINK-4334</a>] - Shaded Hadoop1 jar not fully excluded in Quickstart +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4341">FLINK-4341</a>] - Kinesis connector does not emit maximum watermark properly +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4402">FLINK-4402</a>] - Wrong metrics parameter names in documentation +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4409">FLINK-4409</a>] - class conflict between jsr305-1.3.9.jar and flink-shaded-hadoop2-1.1.1.jar +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4411">FLINK-4411</a>] - [py] Chained dual input children are not properly propagated +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4412">FLINK-4412</a>] - [py] Chaining does not properly handle broadcast variables +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4425">FLINK-4425</a>] - &quot;Out Of Memory&quot; during savepoint deserialization +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4454">FLINK-4454</a>] - Lookups for JobManager address in config +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4480">FLINK-4480</a>] - Incorrect link to elastic.co in documentation +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4486">FLINK-4486</a>] - JobManager not fully running when yarn-session.sh finishes +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4488">FLINK-4488</a>] - Prevent cluster shutdown after job execution for non-detached jobs +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4514">FLINK-4514</a>] - ExpiredIteratorException in Kinesis Consumer on long catch-ups to head of stream +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4526">FLINK-4526</a>] - ApplicationClient: remove redundant proxy messages +</li> + +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3866">FLINK-3866</a>] - StringArraySerializer claims type is immutable; shouldn&#39;t +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-3899">FLINK-3899</a>] - Document window processing with Reduce/FoldFunction + WindowFunction +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4302">FLINK-4302</a>] - Add JavaDocs to MetricConfig +</li> +<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4495">FLINK-4495</a>] - Running multiple jobs on yarn (without yarn-session) +</li> +</ul> + +</description> +<pubDate>Mon, 05 Sep 2016 11:00:00 +0200</pubDate> +<link>http://flink.apache.org/news/2016/09/05/release-1.1.2.html</link> +<guid isPermaLink="true">/news/2016/09/05/release-1.1.2.html</guid> +</item> + +<item> +<title>Flink Forward 2016: Announcing Schedule, Keynotes, and Panel Discussion</title> +<description><p>An update for the Flink community: the <a href="http://flink-forward.org/kb_day/day-1/">Flink Forward 2016 schedule</a> is now available online. This year's event will include 2 days of talks from stream processing experts at Google, MapR, Alibaba, Netflix, Cloudera, and more. Following the talks is a full day of hands-on Flink training.</p> + +<p>Ted Dunning has been announced as a keynote speaker at the event. Ted is the VP of Incubator at <a href="http://www.apache.org">Apache Software Foundation</a>, the Chief Application Architect at <a href="http://www.mapr.com">MapR Technologies</a>, and a mentor on many recent projects. He'll present <a href="http://flink-forward.org/kb_sessions/keynote-tba/">"How Can We Take Flink Forward?"</a> on the second day of the conference.</p> + +<p>Following Ted's keynote there will be a panel discussion on <a href="http://flink-forward.org/kb_sessions/panel-large-scale-streaming-in-production/">"Large Scale Streaming in Production"</a>. As stream processing systems become more mainstream, companies are looking to empower their users to take advantage of this technology. We welcome leading stream processing experts Xiaowei Jiang <a href="http://www.alibaba.com">(Alibaba)</a>, Monal Daxini <a href="http://www.netflix.com">(Netflix)</a>, Maxim Fateev <a href="http://www.uber.com">(Uber)</a>, and Ted Dunning <a href="http://www.mapr.com">(MapR Technologies)</a> on stage to talk about the challenges they have faced and the solutions they have discovered while implementing stream processing systems at very large scale. The panel will be moderated by Jamie Grier <a href="http://www.data-artisan s.com">(data Artisans)</a>.</p> + +<p>The welcome keynote on Monday, September 12, will be presented by data Artisans' co-founders Kostas Tzoumas and Stephan Ewen. They will talk about <a href="http://flink-forward.org/kb_sessions/keynote-tba-2/">"The maturing data streaming ecosystem and Apache Flinkâs accelerated growth"</a>. In this talk, Kostas and Stephan discuss several large-scale stream processing use cases that the data Artisans team has seen over the past year.</p> + +<p>And one more recent addition to the program: Maxim Fateev of Uber will present <a href="http://flink-forward.org/kb_sessions/beyond-the-watermark-on-demand-backfilling-in-flink/">"Beyond the Watermark: On-Demand Backfilling in Flink"</a>. Flinkâs time-progress model is built around a single watermark, which is incompatible with Uberâs business need for generating aggregates retroactively. Maxim's talk covers Uber's solution for on-demand backfilling.</p> + +<p>We hope to see many community members at Flink Forward 2016. Registration is available online: <a href="http://flink-forward.org/registration/">flink-forward.org/registration</a> +</p> +</description> +<pubDate>Wed, 24 Aug 2016 11:00:00 +0200</pubDate> +<link>http://flink.apache.org/news/2016/08/24/ff16-keynotes-panels.html</link> +<guid isPermaLink="true">/news/2016/08/24/ff16-keynotes-panels.html</guid> +</item> + +<item> +<title>Flink 1.1.1 Released</title> +<description><p>Today, the Flink community released Flink version 1.1.1.</p> + +<p>The Maven artifacts published on Maven central for 1.1.0 had a Hadoop dependency issue: No Hadoop 1 specific version (with version 1.1.0-hadoop1) was deployed and 1.1.0 artifacts have a dependency on Hadoop 1 instead of Hadoop 2.</p> + +<p>This was fixed with this release and we <strong>highly recommend</strong> all users to use this version of Flink by bumping your Flink dependencies to version 1.1.1:</p> + +<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span> + <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span> + <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span> + <span class="nt">&lt;version&gt;</span>1.1.1<span class="nt">&lt;/version&gt;</span> +<span class="nt">&lt;/dependency&gt;</span> +<span class="nt">&lt;dependency&gt;</span> + <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span> + <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.10<span class="nt">&lt;/artifactId&gt;</span> + <span class="nt">&lt;version&gt;</span>1.1.1<span class="nt">&lt;/version&gt;</span> +<span class="nt">&lt;/dependency&gt;</span> +<span class="nt">&lt;dependency&gt;</span> + <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span> + <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.10<span class="nt">&lt;/artifactId&gt;</span> + <span class="nt">&lt;version&gt;</span>1.1.1<span class="nt">&lt;/version&gt;</span> +<span class="nt">&lt;/dependency&gt;</span></code></pre></div> + +<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p> +</description> +<pubDate>Thu, 11 Aug 2016 11:00:00 +0200</pubDate> +<link>http://flink.apache.org/news/2016/08/11/release-1.1.1.html</link> +<guid isPermaLink="true">/news/2016/08/11/release-1.1.1.html</guid> +</item> + +<item> +<title>Announcing Apache Flink 1.1.0</title> +<description><div class="alert alert-success"><strong>Important</strong>: The Maven artifacts published with version 1.1.0 on Maven central have a Hadoop dependency issue. It is highly recommended to use <strong>1.1.1</strong> or <strong>1.1.1-hadoop1</strong> as the Flink version.</div> + +<p>The Apache Flink community is pleased to announce the availability of Flink 1.1.0.</p> + +<p>This release is the first major release in the 1.X.X series of releases, which maintains API compatibility with 1.0.0. This means that your applications written against stable APIs of Flink 1.0.0 will compile and run with Flink 1.1.0. 95 contributors provided bug fixes, improvements, and new features such that in total more than 450 JIRA issues could be resolved. See the <a href="/blog/release_1.1.0-changelog.html">complete changelog</a> for more details.</p> + +<p><strong>We encourage everyone to <a href="http://flink.apache.org/downloads.html">download the release</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/">check out the documentation</a>. Feedback through the Flink <a href="http://flink.apache.org/community.html#mailing-lists">mailing lists</a> is, as always, very welcome!</strong></p> + +<p>Some highlights of the release are listed in the following sections.</p> + +<h2 id="connectors">Connectors</h2> + +<p>The <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/connectors/index.html">streaming connectors</a> are a major part of Flinkâs DataStream API. This release adds support for new external systems and further improves on the available connectors.</p> + +<h3 id="continuous-file-system-sources">Continuous File System Sources</h3> + +<p>A frequently requested feature for Flink 1.0 was to be able to monitor directories and process files continuously. Flink 1.1 now adds support for this via <code>FileProcessingMode</code>s:</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">stream</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="na">readFile</span><span class="o">(</span> + <span class="n">textInputFormat</span><span class="o">,</span> + <span class="s">&quot;hdfs:///file-path&quot;</span><span class="o">,</span> + <span class="n">FileProcessingMode</span><span class="o">.</span><span class="na">PROCESS_CONTINUOUSLY</span><span class="o">,</span> + <span class="mi">5000</span><span class="o">,</span> <span class="c1">// monitoring interval (millis)</span> + <span class="n">FilePathFilter</span><span class="o">.</span><span class="na">createDefaultFilter</span><span class="o">());</span> <span class="c1">// file path filter</span></code></pre></div> + +<p>This will monitor <code>hdfs:///file-path</code> every <code>5000</code> milliseconds. Check out the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/index.html#data-sources">DataSource documentation for more details</a>.</p> + +<h3 id="kinesis-source-and-sink">Kinesis Source and Sink</h3> + +<p>Flink 1.1 adds a Kinesis connector for both consuming (<code>FlinkKinesisConsumer</code>) from and producing (<code>FlinkKinesisProduer</code>) to <a href="https://aws.amazon.com/kinesis/">Amazon Kinesis Streams</a>, which is a managed service purpose-built to make it easy to work with streaming data on AWS.</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">kinesis</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span> + <span class="k">new</span> <span class="n">FlinkKinesisConsumer</span><span class="o">&lt;&gt;(</span><span class="s">&quot;stream-name&quot;</span><span class="o">,</span> <span class="n">schema</span><span class="o">,</span> <span class="n">config</span><span class="o">));</span></code></pre></div> + +<p>Check out the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/connectors/kinesis.html">Kinesis connector documentation for more details</a>.</p> + +<h3 id="cassandra-sink">Cassandra Sink</h3> + +<p>The <a href="http://wiki.apache.org/cassandra/GettingStarted">Apache Cassandra</a> sink allows you to write from Flink to Cassandra. Flink can provide exactly-once guarantees if the query is idempotent, meaning it can be applied multiple times without changing the result.</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">CassandraSink</span><span class="o">.</span><span class="na">addSink</span><span class="o">(</span><span class="n">input</span><span class="o">)</span></code></pre></div> + +<p>Check out the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/connectors/cassandra.html">Cassandra Sink documentation for more details</a>.</p> + +<h2 id="table-api-and-sql">Table API and SQL</h2> + +<p>The Table API is a SQL-like expression language for relational stream and batch processing that can be easily embedded in Flinkâs DataSet and DataStream APIs (for both Java and Scala).</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">Table</span> <span class="n">custT</span> <span class="o">=</span> <span class="n">tableEnv</span> + <span class="o">.</span><span class="na">toTable</span><span class="o">(</span><span class="n">custDs</span><span class="o">,</span> <span class="s">&quot;name, zipcode&quot;</span><span class="o">)</span> + <span class="o">.</span><span class="na">where</span><span class="o">(</span><span class="s">&quot;zipcode = &#39;12345&#39;&quot;</span><span class="o">)</span> + <span class="o">.</span><span class="na">select</span><span class="o">(</span><span class="s">&quot;name&quot;</span><span class="o">)</span></code></pre></div> + +<p>An initial version of this API was already available in Flink 1.0. For Flink 1.1, the community put a lot of work into reworking the architecture of the Table API and integrating it with <a href="https://calcite.apache.org">Apache Calcite</a>.</p> + +<p>In this first version, SQL (and Table API) queries on streams are limited to selection, filter, and union operators. Compared to Flink 1.0, the revised Table API supports many more scalar functions and is able to read tables from external sources and write them back to external sinks.</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">Table</span> <span class="n">result</span> <span class="o">=</span> <span class="n">tableEnv</span><span class="o">.</span><span class="na">sql</span><span class="o">(</span> + <span class="s">&quot;SELECT STREAM product, amount FROM Orders WHERE product LIKE &#39;%Rubber%&#39;&quot;</span><span class="o">);</span></code></pre></div> +<p>A more detailed introduction can be found in the <a href="http://flink.apache.org/news/2016/05/24/stream-sql.html">Flink blog</a> and the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/table.html">Table API documentation</a>.</p> + +<h2 id="datastream-api">DataStream API</h2> + +<p>The DataStream API now exposes <strong>session windows</strong> and <strong>allowed lateness</strong> as first-class citizens.</p> + +<h3 id="session-windows">Session Windows</h3> + +<p>Session windows are ideal for cases where the window boundaries need to adjust to the incoming data. This enables you to have windows that start at individual points in time for each key and that end once there has been a <em>certain period of inactivity</em>. The configuration parameter is the session gap that specifies how long to wait for new data before considering a session as closed.</p> + +<center> +<img src="/img/blog/session-windows.svg" style="height:400px" /> +</center> + +<div class="highlight"><pre><code class="language-java"><span class="n">input</span><span class="o">.</span><span class="na">keyBy</span><span class="o">(&lt;</span><span class="n">key</span> <span class="n">selector</span><span class="o">&gt;)</span> + <span class="o">.</span><span class="na">window</span><span class="o">(</span><span class="n">EventTimeSessionWindows</span><span class="o">.</span><span class="na">withGap</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">minutes</span><span class="o">(</span><span class="mi">10</span><span class="o">)))</span> + <span class="o">.&lt;</span><span class="n">windowed</span> <span class="n">transformation</span><span class="o">&gt;(&lt;</span><span class="n">window</span> <span class="n">function</span><span class="o">&gt;);</span></code></pre></div> + +<h3 id="support-for-late-elements">Support for Late Elements</h3> + +<p>You can now specify how a windowed transformation should deal with late elements and how much lateness is allowed. The parameter for this is called <em>allowed lateness</em>. This specifies by how much time elements can be late.</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">input</span><span class="o">.</span><span class="na">keyBy</span><span class="o">(&lt;</span><span class="n">key</span> <span class="n">selector</span><span class="o">&gt;).</span><span class="na">window</span><span class="o">(&lt;</span><span class="n">window</span> <span class="n">assigner</span><span class="o">&gt;)</span> + <span class="o">.</span><span class="na">allowedLateness</span><span class="o">(&lt;</span><span class="n">time</span><span class="o">&gt;)</span> + <span class="o">.&lt;</span><span class="n">windowed</span> <span class="n">transformation</span><span class="o">&gt;(&lt;</span><span class="n">window</span> <span class="n">function</span><span class="o">&gt;);</span></code></pre></div> + +<p>Elements that arrive within the allowed lateness are still put into windows and are considered when computing window results. If elements arrive after the allowed lateness they will be dropped. Flink will also make sure that any state held by the windowing operation is garbage collected once the watermark passes the end of a window plus the allowed lateness.</p> + +<p>Check out the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/windows.html">Windows documentation for more details</a>.</p> + +<h2 id="scala-api-for-complex-event-processing-cep">Scala API for Complex Event Processing (CEP)</h2> + +<p>Flink 1.0 added the initial version of the CEP library. The core of the library is a Pattern API, which allows you to easily specify patterns to match against in your event stream. While in Flink 1.0 this API was only available for Java, Flink 1.1. now exposes the same API for Scala, allowing you to specify your event patterns in a more concise manner.</p> + +<p>A more detailed introduction can be found in the <a href="http://flink.apache.org/news/2016/04/06/cep-monitoring.html">Flink blog</a> and the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/libs/cep.html">CEP documentation</a>.</p> + +<h2 id="graph-generators-and-new-gelly-library-algorithms">Graph generators and new Gelly library algorithms</h2> + +<p>This release includes many enhancements and new features for graph processing. Gelly now provides a collection of scalable graph generators for common graph types, such as complete, cycle, grid, hypercube, and RMat graphs. A variety of new graph algorithms have been added to the Gelly library, including Global and Local Clustering Coefficient, HITS, and similarity measures (Jaccard and Adamic-Adar).</p> + +<p>For a full list of new graph processing features, check out the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/batch/libs/gelly.html">Gelly documentation</a>.</p> + +<h2 id="metrics">Metrics</h2> + +<p>Flinkâs new metrics system allows you to easily gather and expose metrics from your user application to external systems. You can add counters, gauges, and histograms to your application via the runtime context:</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">Counter</span> <span class="n">counter</span> <span class="o">=</span> <span class="n">getRuntimeContext</span><span class="o">()</span> + <span class="o">.</span><span class="na">getMetricGroup</span><span class="o">()</span> + <span class="o">.</span><span class="na">counter</span><span class="o">(</span><span class="s">&quot;my-counter&quot;</span><span class="o">);</span></code></pre></div> + +<p>All registered metrics will be exposed via reporters. Out of the box, Flinks comes with support for JMX, Ganglia, Graphite, and statsD. In addition to your custom metrics, Flink exposes many internal metrics like checkpoint sizes and JVM stats.</p> + +<p>Check out the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/metrics.html">Metrics documentation for more details</a>.</p> + +<h2 id="list-of-contributors">List of Contributors</h2> + +<p>The following 95 people contributed to this release:</p> + +<ul> + <li>Abdullah Ozturk</li> + <li>Ajay Bhat</li> + <li>Alexey Savartsov</li> + <li>Aljoscha Krettek</li> + <li>Andrea Sella</li> + <li>Andrew Palumbo</li> + <li>Chenguang He</li> + <li>Chiwan Park</li> + <li>David Moravek</li> + <li>Dominik Bruhn</li> + <li>Dyana Rose</li> + <li>Fabian Hueske</li> + <li>Flavio Pompermaier</li> + <li>Gabor Gevay</li> + <li>Gabor Horvath</li> + <li>Geoffrey Mon</li> + <li>Gordon Tai</li> + <li>Greg Hogan</li> + <li>Gyula Fora</li> + <li>Henry Saputra</li> + <li>Ignacio N. Lucero Ascencio</li> + <li>Igor Berman</li> + <li>Ismaël MejÃa</li> + <li>Ivan Mushketyk</li> + <li>Jark Wu</li> + <li>Jiri Simsa</li> + <li>Jonas Traub</li> + <li>Josh</li> + <li>Joshi</li> + <li>Joshua Herman</li> + <li>Ken Krugler</li> + <li>Konstantin Knauf</li> + <li>Lasse Dalegaard</li> + <li>Li Fanxi</li> + <li>MaBiao</li> + <li>Mao Wei</li> + <li>Mark Reddy</li> + <li>Martin Junghanns</li> + <li>Martin Liesenberg</li> + <li>Maximilian Michels</li> + <li>Michal Fijolek</li> + <li>Márton Balassi</li> + <li>Nathan Howell</li> + <li>Niels Basjes</li> + <li>Niels Zeilemaker</li> + <li>Phetsarath, Sourigna</li> + <li>Robert Metzger</li> + <li>Scott Kidder</li> + <li>Sebastian Klemke</li> + <li>Shahin</li> + <li>Shannon Carey</li> + <li>Shannon Quinn</li> + <li>Stefan Richter</li> + <li>Stefano Baghino</li> + <li>Stefano Bortoli</li> + <li>Stephan Ewen</li> + <li>Steve Cosenza</li> + <li>Sumit Chawla</li> + <li>Tatu Saloranta</li> + <li>Tianji Li</li> + <li>Till Rohrmann</li> + <li>Todd Lisonbee</li> + <li>Tony Baines</li> + <li>Trevor Grant</li> + <li>Ufuk Celebi</li> + <li>Vasudevan</li> + <li>Yijie Shen</li> + <li>Zack Pierce</li> + <li>Zhai Jia</li> + <li>chengxiang li</li> + <li>chobeat</li> + <li>danielblazevski</li> + <li>dawid</li> + <li>dawidwys</li> + <li>eastcirclek</li> + <li>erli ding</li> + <li>gallenvara</li> + <li>kl0u</li> + <li>mans2singh</li> + <li>markreddy</li> + <li>mjsax</li> + <li>nikste</li> + <li>omaralvarez</li> + <li>philippgrulich</li> + <li>ramkrishna</li> + <li>sahitya-pavurala</li> + <li>samaitra</li> + <li>smarthi</li> + <li>spkavuly</li> + <li>subhankar</li> + <li>twalthr</li> + <li>vasia</li> + <li>xueyan.li</li> + <li>zentol</li> + <li>å«ä¹</li> +</ul> +</description> +<pubDate>Mon, 08 Aug 2016 15:00:00 +0200</pubDate> +<link>http://flink.apache.org/news/2016/08/08/release-1.1.0.html</link> +<guid isPermaLink="true">/news/2016/08/08/release-1.1.0.html</guid> +</item> + +<item> +<title>Stream Processing for Everyone with SQL and Apache Flink</title> +<description><p>The capabilities of open source systems for distributed stream processing have evolved significantly over the last years. Initially, the first systems in the field (notably <a href="https://storm.apache.org">Apache Storm</a>) provided low latency processing, but were limited to at-least-once guarantees, processing-time semantics, and rather low-level APIs. Since then, several new systems emerged and pushed the state of the art of open source stream processing in several dimensions. Today, users of Apache Flink or <a href="https://beam.incubator.apache.org">Apache Beam</a> can use fluent Scala and Java APIs to implement stream processing jobs that operate in event-time with exactly-once semantics at high throughput and low latency.</p> + +<p>In the meantime, stream processing has taken off in the industry. We are witnessing a rapidly growing interest in stream processing which is reflected by prevalent deployments of streaming processing infrastructure such as <a href="https://kafka.apache.org">Apache Kafka</a> and Apache Flink. The increasing number of available data streams results in a demand for people that can analyze streaming data and turn it into real-time insights. However, stream data analysis requires a special skill set including knowledge of streaming concepts such as the characteristics of unbounded streams, windows, time, and state as well as the skills to implement stream analysis jobs usually against Java or Scala APIs. People with this skill set are rare and hard to find.</p> + +<p>About six months ago, the Apache Flink community started an effort to add a SQL interface for stream data analysis. SQL is <em>the</em> standard language to access and process data. Everybody who occasionally analyzes data is familiar with SQL. Consequently, a SQL interface for stream data processing will make this technology accessible to a much wider audience. Moreover, SQL support for streaming data will also enable new use cases such as interactive and ad-hoc stream analysis and significantly simplify many applications including stream ingestion and simple transformations. In this blog post, we report on the current status, architectural design, and future plans of the Apache Flink community to implement support for SQL as a language for analyzing data streams.</p> + +<h2 id="where-did-we-come-from">Where did we come from?</h2> + +<p>With the <a href="http://flink.apache.org/news/2015/04/13/release-0.9.0-milestone1.html">0.9.0-milestone1</a> release, Apache Flink added an API to process relational data with SQL-like expressions called the Table API. The central concept of this API is a Table, a structured data set or stream on which relational operations can be applied. The Table API is tightly integrated with the DataSet and DataStream API. A Table can be easily created from a DataSet or DataStream and can also be converted back into a DataSet or DataStream as the following example shows</p> + +<div class="highlight"><pre><code class="language-scala"><span class="k">val</span> <span class="n">execEnv</span> <span class="k">=</span> <span class="nc">ExecutionEnvironment</span><span class="o">.</span><span class="n">getExecutionEnvironment</span> +<span class="k">val</span> <span class="n">tableEnv</span> <span class="k">=</span> <span class="nc">TableEnvironment</span><span class="o">.</span><span class="n">getTableEnvironment</span><span class="o">(</span><span class="n">execEnv</span><span class="o">)</span> + +<span class="c1">// obtain a DataSet from somewhere</span> +<span class="k">val</span> <span class="n">tempData</span><span class="k">:</span> <span class="kt">DataSet</span><span class="o">[(</span><span class="kt">String</span>, <span class="kt">Long</span>, <span class="kt">Double</span><span class="o">)]</span> <span class="k">=</span> + +<span class="c1">// convert the DataSet to a Table</span> +<span class="k">val</span> <span class="n">tempTable</span><span class="k">:</span> <span class="kt">Table</span> <span class="o">=</span> <span class="n">tempData</span><span class="o">.</span><span class="n">toTable</span><span class="o">(</span><span class="n">tableEnv</span><span class="o">,</span> <span class="-Symbol">&#39;location</span><span class="o">,</span> <span class="-Symbol">&#39;time</span><span class="o">,</span> <span class="-Symbol">&#39;tempF</span><span class="o">)</span> +<span class="c1">// compute your result</span> +<span class="k">val</span> <span class="n">avgTempCTable</span><span class="k">:</span> <span class="kt">Table</span> <span class="o">=</span> <span class="n">tempTable</span> + <span class="o">.</span><span class="n">where</span><span class="o">(</span><span class="-Symbol">&#39;location</span><span class="o">.</span><span class="n">like</span><span class="o">(</span><span class="s">&quot;room%&quot;</span><span class="o">))</span> + <span class="o">.</span><span class="n">select</span><span class="o">(</span> + <span class="o">(</span><span class="-Symbol">&#39;time</span> <span class="o">/</span> <span class="o">(</span><span class="mi">3600</span> <span class="o">*</span> <span class="mi">24</span><span class="o">))</span> <span class="n">as</span> <span class="-Symbol">&#39;day</span><span class="o">,</span> + <span class="-Symbol">&#39;Location</span> <span class="n">as</span> <span class="-Symbol">&#39;room</span><span class="o">,</span> + <span class="o">((</span><span class="-Symbol">&#39;tempF</span> <span class="o">-</span> <span class="mi">32</span><span class="o">)</span> <span class="o">*</span> <span class="mf">0.556</span><span class="o">)</span> <span class="n">as</span> <span class="-Symbol">&#39;tempC</span> + <span class="o">)</span> + <span class="o">.</span><span class="n">groupBy</span><span class="o">(</span><span class="-Symbol">&#39;day</span><span class="o">,</span> <span class="-Symbol">&#39;room</span><span class="o">)</span> + <span class="o">.</span><span class="n">select</span><span class="o">(</span><span class="-Symbol">&#39;day</span><span class="o">,</span> <span class="-Symbol">&#39;room</span><span class="o">,</span> <span class="-Symbol">&#39;tempC</span><span class="o">.</span><span class="n">avg</span> <span class="n">as</span> <span class="-Symbol">&#39;avgTempC</span><span class="o">)</span> +<span class="c1">// convert result Table back into a DataSet and print it</span> +<span class="n">avgTempCTable</span><span class="o">.</span><span class="n">toDataSet</span><span class="o">[</span><span class=
<TRUNCATED>
