This is an automated email from the ASF dual-hosted git repository. fhueske pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit aa25e1f6c9391eb117d0f4d8883b74803de608d8 Author: Fabian Hueske <[email protected]> AuthorDate: Fri May 3 15:44:17 2019 +0200 Rebuild website --- content/2019/05/03/pulsar-flink.html | 350 ++++++++++++++++++++++++++++++ content/blog/feed.xml | 146 +++++++++++++ content/blog/index.html | 38 ++-- content/blog/page2/index.html | 40 ++-- content/blog/page3/index.html | 40 ++-- content/blog/page4/index.html | 40 ++-- content/blog/page5/index.html | 40 ++-- content/blog/page6/index.html | 39 ++-- content/blog/page7/index.html | 42 ++-- content/blog/page8/index.html | 28 +++ content/img/blog/pulsar-flink/image-1.png | Bin 0 -> 71491 bytes content/img/blog/pulsar-flink/image-2.png | Bin 0 -> 180408 bytes content/img/blog/pulsar-flink/image-3.png | Bin 0 -> 144626 bytes content/img/blog/pulsar-flink/image-4.png | Bin 0 -> 312336 bytes content/img/blog/pulsar-flink/image-5.png | Bin 0 -> 563779 bytes content/index.html | 6 +- content/zh/index.html | 6 +- 17 files changed, 702 insertions(+), 113 deletions(-) diff --git a/content/2019/05/03/pulsar-flink.html b/content/2019/05/03/pulsar-flink.html new file mode 100644 index 0000000..de0ce8e --- /dev/null +++ b/content/2019/05/03/pulsar-flink.html @@ -0,0 +1,350 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> + <title>Apache Flink: When Flink & Pulsar Come Together</title> + <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> + <link rel="icon" href="/favicon.ico" type="image/x-icon"> + + <!-- Bootstrap --> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css";> + <link rel="stylesheet" href="/css/flink.css"> + <link rel="stylesheet" href="/css/syntax.css"> + + <!-- Blog RSS feed --> + <link href="/blog/feed.xml" rel="alternate" type="application/rss+xml" title="Apache Flink Blog: RSS feed" /> + + <!-- jQuery (necessary for Bootstrap's JavaScript plugins) --> + <!-- We need to load Jquery in the header for custom google analytics event tracking--> + <script src="/js/jquery.min.js"></script> + + <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> + <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> + <!--[if lt IE 9]> + <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js";></script> + <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js";></script> + <![endif]--> + </head> + <body> + + + <!-- Main content. --> + <div class="container"> + <div class="row"> + + + <div id="sidebar" class="col-sm-3"> + + +<!-- Top navbar. --> + <nav class="navbar navbar-default"> + <!-- The logo. --> + <div class="navbar-header"> + <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <div class="navbar-logo"> + <a href="/"> + <img alt="Apache Flink" src="/img/flink-header-logo.svg" width="147px" height="73px"> + </a> + </div> + </div><!-- /.navbar-header --> + + <!-- The navigation links. --> + <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1"> + <ul class="nav navbar-nav navbar-main"> + + <!-- First menu section explains visitors what Flink is --> + + <!-- What is Stream Processing? --> + <!-- + <li><a href="/streamprocessing1.html">What is Stream Processing?</a></li> + --> + + <!-- What is Flink? --> + <li><a href="/flink-architecture.html">What is Apache Flink?</a></li> + + <!-- Use cases --> + <li><a href="/usecases.html">Use Cases</a></li> + + <!-- Powered by --> + <li><a href="/poweredby.html">Powered By</a></li> + + <!-- FAQ --> + <li><a href="/faq.html">FAQ</a></li> + + + <!-- Second menu section aims to support Flink users --> + + <!-- Downloads --> + <li><a href="/downloads.html">Downloads</a></li> + + <!-- Quickstart --> + <li> + <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/quickstart/setup_quickstart.html"; target="_blank">Tutorials <small><span class="glyphicon glyphicon-new-window"></span></small></a> + </li> + + <!-- Documentation --> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8"; target="_blank">1.8 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + <li><a href="https://ci.apache.org/projects/flink/flink-docs-master"; target="_blank">1.9 (Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + </ul> + </li> + + <!-- getting help --> + <li><a href="/gettinghelp.html">Getting Help</a></li> + + <!-- Blog --> + <li><a href="/blog/"><b>Flink Blog</b></a></li> + + + + <!-- Third menu section aim to support community and contributors --> + + <!-- Community --> + <li><a href="/community.html">Community & Project Info</a></li> + + <!-- Roadmap --> + <li><a href="/roadmap.html">Roadmap</a></li> + + <!-- Contribute --> + <li><a href="/how-to-contribute.html">How to Contribute</a></li> + + <!-- GitHub --> + <li> + <a href="https://github.com/apache/flink"; target="_blank">Flink on GitHub <small><span class="glyphicon glyphicon-new-window"></span></small></a> + </li> + + + + <!-- Language Switcher --> + <li> + + + <a href="/zh/2019/05/03/pulsar-flink.html">中文版</a> + + + </li> + + </ul> + + <ul class="nav navbar-nav navbar-bottom"> + <hr /> + + <!-- Twitter --> + <li><a href="https://twitter.com/apacheflink"; target="_blank">@ApacheFlink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + + <!-- Visualizer --> + <li class=" hidden-md hidden-sm"><a href="/visualizer/" target="_blank">Plan Visualizer <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + + </ul> + </div><!-- /.navbar-collapse --> + </nav> + + </div> + <div class="col-sm-9"> + <div class="row-fluid"> + <div class="col-sm-12"> + <div class="row"> + <h1>When Flink & Pulsar Come Together</h1> + + <article> + <p>03 May 2019 Sijie Guo (<a href="https://twitter.com/sijieg";>@sijieg</a>)</p> + +<p>The open source data technology frameworks <a href="https://flink.apache.org/";>Apache Flink</a> and <a href="https://pulsar.apache.org/en/";>Apache Pulsar</a> can integrate in different ways to provide elastic data processing at large scale. I recently gave a talk at <a href="https://www.flink-forward.org/";>Flink Forward</a> San Francisco 2019 and presented some of the integrations between the two frameworks for batch and streaming applications. In this post, I will give a short introd [...] + +<h2 id="a-brief-introduction-to-apache-pulsar">A brief introduction to Apache Pulsar</h2> + +<p><a href="https://pulsar.apache.org/en/";>Apache Pulsar</a> is an open-source distributed pub-sub messaging system under the stewardship of the <a href="https://www.apache.org/";>Apache Software Foundation</a>. Pulsar is a multi-tenant, high-performance solution for server-to-server messaging including multiple features such as native support for multiple clusters in a Pulsar instance, with seamless <a href="https://pulsar.apache.org/docs/en/administration-geo";>geo-replication</a> of mes [...] + +<p>The first differentiating factor stems from the fact that although Pulsar provides a flexible pub-sub messaging system it is also backed by durable log storage — hence combining both messaging and storage under one framework. Because of that layered architecture, Pulsar provides instant failure recovery, independent scalability and balance-free cluster expansion.</p> + +<p>Pulsar’s architecture follows a similar pattern to other pub-sub systems as the framework is organized in topics as the main data entity, with producers sending data to, and consumers receiving data from a topic as shown in the diagram below.</p> + +<center> +<img src="/img/blog/pulsar-flink/image-1.png" width="400px" alt="Pulsar producers and consumers" /> +</center> +<p><br /></p> + +<p>The second differentiator of Pulsar is that the framework is built from the get-go with <a href="https://pulsar.apache.org/docs/en/concepts-multi-tenancy/";>multi-tenancy</a> in mind. What that means is that each Pulsar topic has a hierarchical management structure making the allocation of resources as well as the resource management and coordination between teams efficient and easy. With Pulsar’s multi-tenancy structure, data platform maintainers can onboard new teams with no friction [...] + +<center> +<img src="/img/blog/pulsar-flink/image-2.png" width="640px" alt="Apache Flink and Apache Pulsar" /> +</center> +<p><br /></p> + +<p>Finally, Pulsar’s flexible messaging framework unifies the streaming and queuing data consumption models and provides greater flexibility. As shown in the below diagram, Pulsar holds the data in the topic while multiple teams can consume the data independently depending on their workloads and data consumption patterns.</p> + +<center> +<img src="/img/blog/pulsar-flink/image-3.png" width="640px" alt="Apache Flink and Apache Pulsar" /> +</center> +<p><br /></p> + +<h2 id="pulsars-view-on-data-segmented-data-streams">Pulsar’s view on data: Segmented data streams</h2> + +<p>Apache Flink is a streaming-first computation framework that perceives <a href="https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html";>batch processing as a special case of streaming</a>. Flink’s view on data streams distinguishes batch and stream processing between bounded and unbounded data streams, assuming that for batch workloads the data stream is finite, with a beginning and an end.</p> + +<p>Apache Pulsar has a similar perspective to that of Apache Flink with regards to the data layer. The framework also uses streams as a unified view on all data, while its layered architecture allows traditional pub-sub messaging for streaming workloads and continuous data processing or usage of <em>Segmented Streams</em> and bounded data stream for batch and static workloads.</p> + +<center> +<img src="/img/blog/pulsar-flink/image-4.png" width="640px" alt="Apache Flink and Apache Pulsar" /> +</center> +<p><br /></p> + +<p>With Pulsar, once a producer sends data to a topic, it is partitioned depending on the data traffic and then further segmented under those partitions — using Apache Bookkeeper as segment store — to allow for parallel data processing as illustrated in the diagram below. This allows a combination of traditional pub-sub messaging and distributed parallel computations in one framework.</p> + +<center> +<img src="/img/blog/pulsar-flink/image-5.png" width="640px" alt="Apache Flink and Apache Pulsar" /> +</center> +<p><br /></p> + +<h2 id="when-flink--pulsar-come-together">When Flink + Pulsar come together</h2> + +<p>Apache Flink and Apache Pulsar integrate in multiple ways already. In the following sections, I will present some potential future integrations between the frameworks and share examples of existing ways in which you can utilize the frameworks together.</p> + +<h3 id="potential-integrations">Potential Integrations</h3> + +<p>Pulsar can integrate with Apache Flink in different ways. Some potential integrations include providing support for streaming workloads with the use of <em>Streaming Connectors</em> and support for batch workloads with the use of <em>Batch Source Connectors</em>. Pulsar also comes with native support for schema that can integrate with Flink and provide structured access to the data, for example by using Flink SQL as a way of querying data in Pulsar. Finally, an alternative way of inte [...] + +<p>From an architecture point of view, we can imagine the integration between the two frameworks as one that uses Apache Pulsar for a unified view of the data layer and Apache Flink as a unified computation and data processing framework and API.</p> + +<h3 id="existing-integrations">Existing Integrations</h3> + +<p>Integration between the two frameworks is ongoing and developers can already use Pulsar with Flink in multiple ways. For example, Pulsar can be used as a streaming source and streaming sink in Flink DataStream applications. Developers can ingest data from Pulsar into a Flink job that makes computations and processes real-time data, to then send the data back to a Pulsar topic as a streaming sink. Such an example is shown below:</p> + +<div class="highlight"><pre><code class="language-java"><span class="c1">// create and configure Pulsar consumer</span> +<span class="n">PulsarSourceBuilder</span><span class="o"><</span><span class="n">String</span><span class="o">></span><span class="n">builder</span> <span class="o">=</span> <span class="n">PulsarSourceBuilder</span> + <span class="o">.</span><span class="na">builder</span><span class="o">(</span><span class="k">new</span> <span class="nf">SimpleStringSchema</span><span class="o">())</span> + <span class="o">.</span><span class="na">serviceUrl</span><span class="o">(</span><span class="n">serviceUrl</span><span class="o">)</span> + <span class="o">.</span><span class="na">topic</span><span class="o">(</span><span class="n">inputTopic</span><span class="o">)</span> + <span class="o">.</span><span class="na">subscriptionName</span><span class="o">(</span><span class="n">subscription</span><span class="o">);</span> +<span class="n">SourceFunction</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">src</span> <span class="o">=</span> <span class="n">builder</span><span class="o">.</span><span class="na">build</span><span class="o">();</span> +<span class="c1">// ingest DataStream with Pulsar consumer</span> +<span class="n">DataStream</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">words</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span><span class="n">src</span><span class="o">);</span> + +<span class="c1">// perform computation on DataStream (here a simple WordCount)</span> +<span class="n">DataStream</span><span class="o"><</span><span class="n">WordWithCount</span><span class="o">></span> <span class="n">wc</span> <span class="o">=</span> <span class="n">words</span> + <span class="o">.</span><span class="na">flatMap</span><span class="o">((</span><span class="n">FlatMapFunction</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">WordWithCount</span><span class="o">>)</span> <span class="o">(</span><span class="n">word</span><span class="o">,</span> <span class="n">collector</span><span class="o">)</span> <span class="o">-></span> <span class="o">{</span> + <span class="n">collector</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="k">new</span> <span class="nf">WordWithCount</span><span class="o">(</span><span class="n">word</span><span class="o">,</span> <span class="mi">1</span><span class="o">));</span> + <span class="o">})</span> + <span class="o">.</span><span class="na">returns</span><span class="o">(</span><span class="n">WordWithCount</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> + <span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="s">"word"</span><span class="o">)</span> + <span class="o">.</span><span class="na">timeWindow</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">seconds</span><span class="o">(</span><span class="mi">5</span><span class="o">))</span> + <span class="o">.</span><span class="na">reduce</span><span class="o">((</span><span class="n">ReduceFunction</span><span class="o"><</span><span class="n">WordWithCount</span><span class="o">>)</span> <span class="o">(</span><span class="n">c1</span><span class="o">,</span> <span class="n">c2</span><span class="o">)</span> <span class="o">-></span> + <span class="k">new</span> <span class="nf">WordWithCount</span><span class="o">(</span><span class="n">c1</span><span class="o">.</span><span class="na">word</span><span class="o">,</span> <span class="n">c1</span><span class="o">.</span><span class="na">count</span> <span class="o">+</span> <span class="n">c2</span><span class="o">.</span><span class="na">count</span><span class="o">));</span> + +<span class="c1">// emit result via Pulsar producer</span> +<span class="n">wc</span><span class="o">.</span><span class="na">addSink</span><span class="o">(</span><span class="k">new</span> <span class="n">FlinkPulsarProducer</span><span class="o"><>(</span> + <span class="n">serviceUrl</span><span class="o">,</span> + <span class="n">outputTopic</span><span class="o">,</span> + <span class="k">new</span> <span class="nf">AuthenticationDisabled</span><span class="o">(),</span> + <span class="n">wordWithCount</span> <span class="o">-></span> <span class="n">wordWithCount</span><span class="o">.</span><span class="na">toString</span><span class="o">().</span><span class="na">getBytes</span><span class="o">(</span><span class="n">UTF_8</span><span class="o">),</span> + <span class="n">wordWithCount</span> <span class="o">-></span> <span class="n">wordWithCount</span><span class="o">.</span><span class="na">word</span><span class="o">)</span> +<span class="o">);</span></code></pre></div> + +<p>Another integration between the two frameworks that developers can take advantage of includes using Pulsar as both a streaming source and a streaming table sink for Flink SQL or Table API queries as shown in the example below:</p> + +<div class="highlight"><pre><code class="language-java"><span class="c1">// obtain a DataStream with words</span> +<span class="n">DataStream</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">words</span> <span class="o">=</span> <span class="o">...</span> + +<span class="c1">// register DataStream as Table "words" with two attributes ("word", "ts"). </span> +<span class="c1">// "ts" is an event-time timestamp.</span> +<span class="n">tableEnvironment</span><span class="o">.</span><span class="na">registerDataStream</span><span class="o">(</span><span class="s">"words"</span><span class="o">,</span> <span class="n">words</span><span class="o">,</span> <span class="s">"word, ts.rowtime"</span><span class="o">);</span> + +<span class="c1">// create a TableSink that produces to Pulsar</span> +<span class="n">TableSink</span> <span class="n">sink</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">PulsarJsonTableSink</span><span class="o">(</span> + <span class="n">serviceUrl</span><span class="o">,</span> + <span class="n">outputTopic</span><span class="o">,</span> + <span class="k">new</span> <span class="nf">AuthenticationDisabled</span><span class="o">(),</span> + <span class="n">ROUTING_KEY</span><span class="o">);</span> + +<span class="c1">// register Pulsar TableSink as table "wc"</span> +<span class="n">tableEnvironment</span><span class="o">.</span><span class="na">registerTableSink</span><span class="o">(</span> + <span class="s">"wc"</span><span class="o">,</span> + <span class="n">sink</span><span class="o">.</span><span class="na">configure</span><span class="o">(</span> + <span class="k">new</span> <span class="n">String</span><span class="o">[]{</span><span class="s">"word"</span><span class="o">,</span> <span class="s">"cnt"</span><span class="o">},</span> + <span class="k">new</span> <span class="n">TypeInformation</span><span class="o">[]{</span><span class="n">Types</span><span class="o">.</span><span class="na">STRING</span><span class="o">,</span> <span class="n">Types</span><span class="o">.</span><span class="na">LONG</span><span class="o">}));</span> + +<span class="c1">// count words per 5 seconds and write result to table "wc"</span> +<span class="n">tableEnvironment</span><span class="o">.</span><span class="na">sqlUpdate</span><span class="o">(</span> + <span class="s">"INSERT INTO wc "</span> <span class="o">+</span> + <span class="s">"SELECT word, COUNT(*) AS cnt "</span> <span class="o">+</span> + <span class="s">"FROM words "</span> <span class="o">+</span> + <span class="s">"GROUP BY word, TUMBLE(ts, INTERVAL '5' SECOND)"</span><span class="o">);</span></code></pre></div> + +<p>Finally, Flink integrates with Pulsar for batch workloads as a batch sink where all results get pushed to Pulsar after Apache Flink has completed the computation in a static data set. Such an example is shown below:</p> + +<div class="highlight"><pre><code class="language-java"><span class="c1">// obtain DataSet from arbitrary computation</span> +<span class="n">DataSet</span><span class="o"><</span><span class="n">WordWithCount</span><span class="o">></span> <span class="n">wc</span> <span class="o">=</span> <span class="o">...</span> + +<span class="c1">// create PulsarOutputFormat instance</span> +<span class="n">OutputFormat</span> <span class="n">pulsarOutputFormat</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">PulsarOutputFormat</span><span class="o">(</span> + <span class="n">serviceUrl</span><span class="o">,</span> + <span class="n">topic</span><span class="o">,</span> + <span class="k">new</span> <span class="nf">AuthenticationDisabled</span><span class="o">(),</span> + <span class="n">wordWithCount</span> <span class="o">-></span> <span class="n">wordWithCount</span><span class="o">.</span><span class="na">toString</span><span class="o">().</span><span class="na">getBytes</span><span class="o">());</span> +<span class="c1">// write DataSet to Pulsar</span> +<span class="n">wc</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">pulsarOutputFormat</span><span class="o">);</span></code></pre></div> + +<h2 id="conclusion">Conclusion</h2> + +<p>Both Pulsar and Flink share a similar view on how the data and the computation level of an application can be <em>“streaming-first”</em> with batch as a special case streaming. With Pulsar’s Segmented Streams approach and Flink’s steps to unify batch and stream processing workloads under one framework, there are numerous ways of integrating the two technologies together to provide elastic data processing at massive scale. Subscribe to the <a href="https://flink.apache.org/community.ht [...] + + </article> + </div> + + <div class="row"> + <div id="disqus_thread"></div> + <script type="text/javascript"> + /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */ + var disqus_shortname = 'stratosphere-eu'; // required: replace example with your forum shortname + + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); + </script> + </div> + </div> +</div> + </div> + </div> + + <hr /> + + <div class="row"> + <div class="footer text-center col-sm-12"> + <p>Copyright © 2014-2019 <a href="http://apache.org";>The Apache Software Foundation</a>. All Rights Reserved.</p> + <p>Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.</p> + <p><a href="/privacy-policy.html">Privacy Policy</a> · <a href="/blog/feed.xml">RSS feed</a></p> + </div> + </div> + </div><!-- /.container --> + + <!-- Include all compiled plugins (below), or include individual files as needed --> + <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js";></script> + <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery.matchHeight/0.7.0/jquery.matchHeight-min.js";></script> + <script src="/js/codetabs.js"></script> + <script src="/js/stickysidebar.js"></script> + + <!-- Google Analytics --> + <script> + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + + ga('create', 'UA-52545728-1', 'auto'); + ga('send', 'pageview'); + </script> + </body> +</html> diff --git a/content/blog/feed.xml b/content/blog/feed.xml index df8c4e2..5ea6436 100644 --- a/content/blog/feed.xml +++ b/content/blog/feed.xml @@ -7,6 +7,152 @@ <atom:link href="https://flink.apache.org/blog/feed.xml"; rel="self" type="application/rss+xml" /> <item> +<title>When Flink & Pulsar Come Together</title> +<description><p>The open source data technology frameworks <a href="https://flink.apache.org/">Apache Flink</a> and <a href="https://pulsar.apache.org/en/">Apache Pulsar</a> can integrate in different ways to provide elastic data processing at large scale. I recently gave a talk at <a href="https://www.flink-forward.org/">Flink Forward</a> San Francisco 2019 and presented some of the integrations between the two fram [...] + +<h2 id="a-brief-introduction-to-apache-pulsar">A brief introduction to Apache Pulsar</h2> + +<p><a href="https://pulsar.apache.org/en/">Apache Pulsar</a> is an open-source distributed pub-sub messaging system under the stewardship of the <a href="https://www.apache.org/">Apache Software Foundation</a>. Pulsar is a multi-tenant, high-performance solution for server-to-server messaging including multiple features such as native support for multiple clusters in a Pulsar instance, with seamless <a href="https://pulsar.apache. [...] + +<p>The first differentiating factor stems from the fact that although Pulsar provides a flexible pub-sub messaging system it is also backed by durable log storage — hence combining both messaging and storage under one framework. Because of that layered architecture, Pulsar provides instant failure recovery, independent scalability and balance-free cluster expansion.</p> + +<p>Pulsar’s architecture follows a similar pattern to other pub-sub systems as the framework is organized in topics as the main data entity, with producers sending data to, and consumers receiving data from a topic as shown in the diagram below.</p> + +<center> +<img src="/img/blog/pulsar-flink/image-1.png" width="400px" alt="Pulsar producers and consumers" /> +</center> +<p><br /></p> + +<p>The second differentiator of Pulsar is that the framework is built from the get-go with <a href="https://pulsar.apache.org/docs/en/concepts-multi-tenancy/">multi-tenancy</a> in mind. What that means is that each Pulsar topic has a hierarchical management structure making the allocation of resources as well as the resource management and coordination between teams efficient and easy. With Pulsar’s multi-tenancy structure, data platform maintainers can onboar [...] + +<center> +<img src="/img/blog/pulsar-flink/image-2.png" width="640px" alt="Apache Flink and Apache Pulsar" /> +</center> +<p><br /></p> + +<p>Finally, Pulsar’s flexible messaging framework unifies the streaming and queuing data consumption models and provides greater flexibility. As shown in the below diagram, Pulsar holds the data in the topic while multiple teams can consume the data independently depending on their workloads and data consumption patterns.</p> + +<center> +<img src="/img/blog/pulsar-flink/image-3.png" width="640px" alt="Apache Flink and Apache Pulsar" /> +</center> +<p><br /></p> + +<h2 id="pulsars-view-on-data-segmented-data-streams">Pulsar’s view on data: Segmented data streams</h2> + +<p>Apache Flink is a streaming-first computation framework that perceives <a href="https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html">batch processing as a special case of streaming</a>. Flink’s view on data streams distinguishes batch and stream processing between bounded and unbounded data streams, assuming that for batch workloads the data stream is finite, with a beginning and an end.</p> + +<p>Apache Pulsar has a similar perspective to that of Apache Flink with regards to the data layer. The framework also uses streams as a unified view on all data, while its layered architecture allows traditional pub-sub messaging for streaming workloads and continuous data processing or usage of <em>Segmented Streams</em> and bounded data stream for batch and static workloads.</p> + +<center> +<img src="/img/blog/pulsar-flink/image-4.png" width="640px" alt="Apache Flink and Apache Pulsar" /> +</center> +<p><br /></p> + +<p>With Pulsar, once a producer sends data to a topic, it is partitioned depending on the data traffic and then further segmented under those partitions — using Apache Bookkeeper as segment store — to allow for parallel data processing as illustrated in the diagram below. This allows a combination of traditional pub-sub messaging and distributed parallel computations in one framework.</p> + +<center> +<img src="/img/blog/pulsar-flink/image-5.png" width="640px" alt="Apache Flink and Apache Pulsar" /> +</center> +<p><br /></p> + +<h2 id="when-flink--pulsar-come-together">When Flink + Pulsar come together</h2> + +<p>Apache Flink and Apache Pulsar integrate in multiple ways already. In the following sections, I will present some potential future integrations between the frameworks and share examples of existing ways in which you can utilize the frameworks together.</p> + +<h3 id="potential-integrations">Potential Integrations</h3> + +<p>Pulsar can integrate with Apache Flink in different ways. Some potential integrations include providing support for streaming workloads with the use of <em>Streaming Connectors</em> and support for batch workloads with the use of <em>Batch Source Connectors</em>. Pulsar also comes with native support for schema that can integrate with Flink and provide structured access to the data, for example by using Flink SQL as a way of querying data in Pulsar. Final [...] + +<p>From an architecture point of view, we can imagine the integration between the two frameworks as one that uses Apache Pulsar for a unified view of the data layer and Apache Flink as a unified computation and data processing framework and API.</p> + +<h3 id="existing-integrations">Existing Integrations</h3> + +<p>Integration between the two frameworks is ongoing and developers can already use Pulsar with Flink in multiple ways. For example, Pulsar can be used as a streaming source and streaming sink in Flink DataStream applications. Developers can ingest data from Pulsar into a Flink job that makes computations and processes real-time data, to then send the data back to a Pulsar topic as a streaming sink. Such an example is shown below:</p> + +<div class="highlight"><pre><code class="language-java"><span class="c1">// create and configure Pulsar consumer</span> +<span class="n">PulsarSourceBuilder</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">builder</span> <span class="o">=</span> <span class="n">PulsarSourceBuilder</span> + <span class="o">.</span><span class="na">builder</span><span class="o">(</span><span class="k">new</span> <span class="nf">SimpleStringSchema</span><span class="o">())</span> + <span class="o">.</span><span class="na">serviceUrl</span><span class="o">(</span><span class="n">serviceUrl</span><span class="o">)</span> + <span class="o">.</span><span class="na">topic</span><span class="o">(</span><span class="n">inputTopic</span><span class="o">)</span> + <span class="o">.</span><span class="na">subscriptionName</span><span class="o">(</span><span class="n">subscription</span><span class="o">);</span> +<span class="n">SourceFunction</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">src</span> <span class="o">=</span> <span class="n">builder</span><span class="o">.</span><span class="na">build</span><span class="o"&g [...] +<span class="c1">// ingest DataStream with Pulsar consumer</span> +<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">words</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o"> [...] + +<span class="c1">// perform computation on DataStream (here a simple WordCount)</span> +<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">WordWithCount</span><span class="o">&gt;</span> <span class="n">wc</span> <span class="o">=</span> <span class="n">words</span> + <span class="o">.</span><span class="na">flatMap</span><span class="o">((</span><span class="n">FlatMapFunction</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">WordWithCount</span><span class="o">&gt;)</span> <span class=&quo [...] + <span class="n">collector</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="k">new</span> <span class="nf">WordWithCount</span><span class="o">(</span><span class="n">word</span><span class="o">,</span> <span class="mi">1&l [...] + <span class="o">})</span> + <span class="o">.</span><span class="na">returns</span><span class="o">(</span><span class="n">WordWithCount</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> + <span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="s">&quot;word&quot;</span><span class="o">)</span> + <span class="o">.</span><span class="na">timeWindow</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">seconds</span><span class="o">(</span><span class="mi">5</span><span class="o">))</span> + <span class="o">.</span><span class="na">reduce</span><span class="o">((</span><span class="n">ReduceFunction</span><span class="o">&lt;</span><span class="n">WordWithCount</span><span class="o">&gt;)</span> <span class="o">(</span><span class="n">c1</span><span class="o&quo [...] + <span class="k">new</span> <span class="nf">WordWithCount</span><span class="o">(</span><span class="n">c1</span><span class="o">.</span><span class="na">word</span><span class="o">,</span> <span class="n">c1</span><span class="o">.</span><span class="na">count</span& [...] + +<span class="c1">// emit result via Pulsar producer</span> +<span class="n">wc</span><span class="o">.</span><span class="na">addSink</span><span class="o">(</span><span class="k">new</span> <span class="n">FlinkPulsarProducer</span><span class="o">&lt;&gt;(</span> + <span class="n">serviceUrl</span><span class="o">,</span> + <span class="n">outputTopic</span><span class="o">,</span> + <span class="k">new</span> <span class="nf">AuthenticationDisabled</span><span class="o">(),</span> + <span class="n">wordWithCount</span> <span class="o">-&gt;</span> <span class="n">wordWithCount</span><span class="o">.</span><span class="na">toString</span><span class="o">().</span><span class="na">getBytes</span><span class="o">(</span><span class="n">UTF_8</span><span class=&quo [...] + <span class="n">wordWithCount</span> <span class="o">-&gt;</span> <span class="n">wordWithCount</span><span class="o">.</span><span class="na">word</span><span class="o">)</span> +<span class="o">);</span></code></pre></div> + +<p>Another integration between the two frameworks that developers can take advantage of includes using Pulsar as both a streaming source and a streaming table sink for Flink SQL or Table API queries as shown in the example below:</p> + +<div class="highlight"><pre><code class="language-java"><span class="c1">// obtain a DataStream with words</span> +<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">words</span> <span class="o">=</span> <span class="o">...</span> + +<span class="c1">// register DataStream as Table &quot;words&quot; with two attributes (&quot;word&quot;, &quot;ts&quot;). </span> +<span class="c1">// &quot;ts&quot; is an event-time timestamp.</span> +<span class="n">tableEnvironment</span><span class="o">.</span><span class="na">registerDataStream</span><span class="o">(</span><span class="s">&quot;words&quot;</span><span class="o">,</span> <span class="n">words</span><span class="o">,</span> <span class="s">&quot;word, ts.rowtime&am [...] + +<span class="c1">// create a TableSink that produces to Pulsar</span> +<span class="n">TableSink</span> <span class="n">sink</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">PulsarJsonTableSink</span><span class="o">(</span> + <span class="n">serviceUrl</span><span class="o">,</span> + <span class="n">outputTopic</span><span class="o">,</span> + <span class="k">new</span> <span class="nf">AuthenticationDisabled</span><span class="o">(),</span> + <span class="n">ROUTING_KEY</span><span class="o">);</span> + +<span class="c1">// register Pulsar TableSink as table &quot;wc&quot;</span> +<span class="n">tableEnvironment</span><span class="o">.</span><span class="na">registerTableSink</span><span class="o">(</span> + <span class="s">&quot;wc&quot;</span><span class="o">,</span> + <span class="n">sink</span><span class="o">.</span><span class="na">configure</span><span class="o">(</span> + <span class="k">new</span> <span class="n">String</span><span class="o">[]{</span><span class="s">&quot;word&quot;</span><span class="o">,</span> <span class="s">&quot;cnt&quot;</span><span class="o">},</span> + <span class="k">new</span> <span class="n">TypeInformation</span><span class="o">[]{</span><span class="n">Types</span><span class="o">.</span><span class="na">STRING</span><span class="o">,</span> <span class="n">Types</span><span class="o">.</span><span class="na">LONG& [...] + +<span class="c1">// count words per 5 seconds and write result to table &quot;wc&quot;</span> +<span class="n">tableEnvironment</span><span class="o">.</span><span class="na">sqlUpdate</span><span class="o">(</span> + <span class="s">&quot;INSERT INTO wc &quot;</span> <span class="o">+</span> + <span class="s">&quot;SELECT word, COUNT(*) AS cnt &quot;</span> <span class="o">+</span> + <span class="s">&quot;FROM words &quot;</span> <span class="o">+</span> + <span class="s">&quot;GROUP BY word, TUMBLE(ts, INTERVAL &#39;5&#39; SECOND)&quot;</span><span class="o">);</span></code></pre></div> + +<p>Finally, Flink integrates with Pulsar for batch workloads as a batch sink where all results get pushed to Pulsar after Apache Flink has completed the computation in a static data set. Such an example is shown below:</p> + +<div class="highlight"><pre><code class="language-java"><span class="c1">// obtain DataSet from arbitrary computation</span> +<span class="n">DataSet</span><span class="o">&lt;</span><span class="n">WordWithCount</span><span class="o">&gt;</span> <span class="n">wc</span> <span class="o">=</span> <span class="o">...</span> + +<span class="c1">// create PulsarOutputFormat instance</span> +<span class="n">OutputFormat</span> <span class="n">pulsarOutputFormat</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">PulsarOutputFormat</span><span class="o">(</span> + <span class="n">serviceUrl</span><span class="o">,</span> + <span class="n">topic</span><span class="o">,</span> + <span class="k">new</span> <span class="nf">AuthenticationDisabled</span><span class="o">(),</span> + <span class="n">wordWithCount</span> <span class="o">-&gt;</span> <span class="n">wordWithCount</span><span class="o">.</span><span class="na">toString</span><span class="o">().</span><span class="na">getBytes</span><span class="o">());</span> +<span class="c1">// write DataSet to Pulsar</span> +<span class="n">wc</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">pulsarOutputFormat</span><span class="o">);</span></code></pre></div> + +<h2 id="conclusion">Conclusion</h2> + +<p>Both Pulsar and Flink share a similar view on how the data and the computation level of an application can be <em>“streaming-first”</em> with batch as a special case streaming. With Pulsar’s Segmented Streams approach and Flink’s steps to unify batch and stream processing workloads under one framework, there are numerous ways of integrating the two technologies together to provide elastic data processing at massive scale. Subscribe to the <a href="https://fli [...] +</description> +<pubDate>Fri, 03 May 2019 14:00:00 +0200</pubDate> +<link>https://flink.apache.org/2019/05/03/pulsar-flink.html</link> +<guid isPermaLink="true">/2019/05/03/pulsar-flink.html</guid> +</item> + +<item> <title>Apache Flink's Application to Season of Docs</title> <description><p>The Apache Flink community is happy to announce its application to the first edition of <a href="https://developers.google.com/season-of-docs/">Season of Docs</a> by Google. The program is bringing together Open Source projects and technical writers to raise awareness for and improve documentation of Open Source projects. While the community is continuously looking for new contributors to collaborate on our documentation, we would like to take [...] diff --git a/content/blog/index.html b/content/blog/index.html index e4cfd33..df60f88 100644 --- a/content/blog/index.html +++ b/content/blog/index.html @@ -159,6 +159,19 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/2019/05/03/pulsar-flink.html">When Flink & Pulsar Come Together</a></h2> + + <p>03 May 2019 + Sijie Guo (<a href="https://twitter.com/sijieg";>@sijieg</a>)</p> + + <p>Apache Flink and Apache Pulsar are distributed data processing systems. When combined, they offer elastic data processing at large scale. This post describes how Pulsar and Flink can work together to provide a seamless developer experience.</p> + + <p><a href="/2019/05/03/pulsar-flink.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2019/04/17/sod.html">Apache Flink's Application to Season of Docs</a></h2> <p>17 Apr 2019 @@ -289,21 +302,6 @@ for more details.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2018/12/22/release-1.6.3.html">Apache Flink 1.6.3 Released</a></h2> - - <p>22 Dec 2018 - </p> - - <p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.6 series.</p> - -</p> - - <p><a href="/news/2018/12/22/release-1.6.3.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -336,6 +334,16 @@ for more details.</p> <ul id="markdown-toc"> + <li><a href="/2019/05/03/pulsar-flink.html">When Flink & Pulsar Come Together</a></li> + + + + + + + + + <li><a href="/news/2019/04/17/sod.html">Apache Flink's Application to Season of Docs</a></li> diff --git a/content/blog/page2/index.html b/content/blog/page2/index.html index a81f119..02b76e4 100644 --- a/content/blog/page2/index.html +++ b/content/blog/page2/index.html @@ -159,6 +159,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2018/12/22/release-1.6.3.html">Apache Flink 1.6.3 Released</a></h2> + + <p>22 Dec 2018 + </p> + + <p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.6 series.</p> + +</p> + + <p><a href="/news/2018/12/22/release-1.6.3.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2018/12/21/release-1.7.1.html">Apache Flink 1.7.1 Released</a></h2> <p>21 Dec 2018 @@ -295,21 +310,6 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa <hr> - <article> - <h2 class="blog-title"><a href="/news/2018/07/12/release-1.5.1.html">Apache Flink 1.5.1 Released</a></h2> - - <p>12 Jul 2018 - </p> - - <p><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.5 series.</p> - -</p> - - <p><a href="/news/2018/07/12/release-1.5.1.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -342,6 +342,16 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa <ul id="markdown-toc"> + <li><a href="/2019/05/03/pulsar-flink.html">When Flink & Pulsar Come Together</a></li> + + + + + + + + + <li><a href="/news/2019/04/17/sod.html">Apache Flink's Application to Season of Docs</a></li> diff --git a/content/blog/page3/index.html b/content/blog/page3/index.html index e30529b..12e5b7a 100644 --- a/content/blog/page3/index.html +++ b/content/blog/page3/index.html @@ -159,6 +159,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2018/07/12/release-1.5.1.html">Apache Flink 1.5.1 Released</a></h2> + + <p>12 Jul 2018 + </p> + + <p><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.5 series.</p> + +</p> + + <p><a href="/news/2018/07/12/release-1.5.1.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2018/05/25/release-1.5.0.html">Apache Flink 1.5.0 Release Announcement</a></h2> <p>25 May 2018 @@ -292,21 +307,6 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community <hr> - <article> - <h2 class="blog-title"><a href="/news/2017/08/05/release-1.3.2.html">Apache Flink 1.3.2 Released</a></h2> - - <p>05 Aug 2017 - </p> - - <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.3 series.</p> - -</p> - - <p><a href="/news/2017/08/05/release-1.3.2.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -339,6 +339,16 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community <ul id="markdown-toc"> + <li><a href="/2019/05/03/pulsar-flink.html">When Flink & Pulsar Come Together</a></li> + + + + + + + + + <li><a href="/news/2019/04/17/sod.html">Apache Flink's Application to Season of Docs</a></li> diff --git a/content/blog/page4/index.html b/content/blog/page4/index.html index 0c28257..f505a29 100644 --- a/content/blog/page4/index.html +++ b/content/blog/page4/index.html @@ -159,6 +159,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2017/08/05/release-1.3.2.html">Apache Flink 1.3.2 Released</a></h2> + + <p>05 Aug 2017 + </p> + + <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.3 series.</p> + +</p> + + <p><a href="/news/2017/08/05/release-1.3.2.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/features/2017/07/04/flink-rescalable-state.html">A Deep Dive into Rescalable State in Apache Flink</a></h2> <p>04 Jul 2017 by Stefan Richter (<a href="https://twitter.com/";>@StefanRRichter</a>) @@ -286,21 +301,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2016/12/21/release-1.1.4.html">Apache Flink 1.1.4 Released</a></h2> - - <p>21 Dec 2016 - </p> - - <p><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.1 series.</p> - -</p> - - <p><a href="/news/2016/12/21/release-1.1.4.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -333,6 +333,16 @@ <ul id="markdown-toc"> + <li><a href="/2019/05/03/pulsar-flink.html">When Flink & Pulsar Come Together</a></li> + + + + + + + + + <li><a href="/news/2019/04/17/sod.html">Apache Flink's Application to Season of Docs</a></li> diff --git a/content/blog/page5/index.html b/content/blog/page5/index.html index 5ef807e..e1b1eb3 100644 --- a/content/blog/page5/index.html +++ b/content/blog/page5/index.html @@ -159,6 +159,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2016/12/21/release-1.1.4.html">Apache Flink 1.1.4 Released</a></h2> + + <p>21 Dec 2016 + </p> + + <p><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.1 series.</p> + +</p> + + <p><a href="/news/2016/12/21/release-1.1.4.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2016/12/19/2016-year-in-review.html">Apache Flink in 2016: Year in Review</a></h2> <p>19 Dec 2016 by Mike Winters @@ -290,21 +305,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2016/04/14/flink-forward-announce.html">Flink Forward 2016 Call for Submissions Is Now Open</a></h2> - - <p>14 Apr 2016 by Aljoscha Krettek (<a href="https://twitter.com/";>@aljoscha</a>) - </p> - - <p><p>We are happy to announce that the call for submissions for Flink Forward 2016 is now open! The conference will take place September 12-14, 2016 in Berlin, Germany, bringing together the open source stream processing community. Most Apache Flink committers will attend the conference, making it the ideal venue to learn more about the project and its roadmap and connect with the community.</p> - -</p> - - <p><a href="/news/2016/04/14/flink-forward-announce.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -337,6 +337,16 @@ <ul id="markdown-toc"> + <li><a href="/2019/05/03/pulsar-flink.html">When Flink & Pulsar Come Together</a></li> + + + + + + + + + <li><a href="/news/2019/04/17/sod.html">Apache Flink's Application to Season of Docs</a></li> diff --git a/content/blog/page6/index.html b/content/blog/page6/index.html index b877fb7..a8bf38f 100644 --- a/content/blog/page6/index.html +++ b/content/blog/page6/index.html @@ -159,6 +159,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2016/04/14/flink-forward-announce.html">Flink Forward 2016 Call for Submissions Is Now Open</a></h2> + + <p>14 Apr 2016 by Aljoscha Krettek (<a href="https://twitter.com/";>@aljoscha</a>) + </p> + + <p><p>We are happy to announce that the call for submissions for Flink Forward 2016 is now open! The conference will take place September 12-14, 2016 in Berlin, Germany, bringing together the open source stream processing community. Most Apache Flink committers will attend the conference, making it the ideal venue to learn more about the project and its roadmap and connect with the community.</p> + +</p> + + <p><a href="/news/2016/04/14/flink-forward-announce.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2016/04/06/cep-monitoring.html">Introducing Complex Event Processing (CEP) with Apache Flink</a></h2> <p>06 Apr 2016 by Till Rohrmann (<a href="https://twitter.com/";>@stsffap</a>) @@ -286,20 +301,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2015/09/16/off-heap-memory.html">Off-heap Memory in Apache Flink and the curious JIT compiler</a></h2> - - <p>16 Sep 2015 by Stephan Ewen (<a href="https://twitter.com/";>@stephanewen</a>) - </p> - - <p><p>Running data-intensive code in the JVM and making it well-behaved is tricky. Systems that put billions of data objects naively onto the JVM heap face unpredictable OutOfMemoryErrors and Garbage Collection stalls. Of course, you still want to to keep your data in memory as much as possible, for speed and responsiveness of the processing applications. In that context, "off-heap" has become almost something like a magic word to solve these problems.</p> -<p>In this blog post, we will look at how Flink exploits off-heap memory. The feature is part of the upcoming release, but you can try it out with the latest nightly builds. We will also give a few interesting insights into the behavior for Java's JIT compiler for highly optimized methods and loops.</p></p> - - <p><a href="/news/2015/09/16/off-heap-memory.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -332,6 +333,16 @@ <ul id="markdown-toc"> + <li><a href="/2019/05/03/pulsar-flink.html">When Flink & Pulsar Come Together</a></li> + + + + + + + + + <li><a href="/news/2019/04/17/sod.html">Apache Flink's Application to Season of Docs</a></li> diff --git a/content/blog/page7/index.html b/content/blog/page7/index.html index 1c7182d..c8c20b5 100644 --- a/content/blog/page7/index.html +++ b/content/blog/page7/index.html @@ -159,6 +159,20 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2015/09/16/off-heap-memory.html">Off-heap Memory in Apache Flink and the curious JIT compiler</a></h2> + + <p>16 Sep 2015 by Stephan Ewen (<a href="https://twitter.com/";>@stephanewen</a>) + </p> + + <p><p>Running data-intensive code in the JVM and making it well-behaved is tricky. Systems that put billions of data objects naively onto the JVM heap face unpredictable OutOfMemoryErrors and Garbage Collection stalls. Of course, you still want to to keep your data in memory as much as possible, for speed and responsiveness of the processing applications. In that context, "off-heap" has become almost something like a magic word to solve these problems.</p> +<p>In this blog post, we will look at how Flink exploits off-heap memory. The feature is part of the upcoming release, but you can try it out with the latest nightly builds. We will also give a few interesting insights into the behavior for Java's JIT compiler for highly optimized methods and loops.</p></p> + + <p><a href="/news/2015/09/16/off-heap-memory.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2015/09/03/flink-forward.html">Announcing Flink Forward 2015</a></h2> <p>03 Sep 2015 @@ -300,24 +314,6 @@ release is a preview release that contains known issues.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2015/03/02/february-2015-in-flink.html">February 2015 in the Flink community</a></h2> - - <p>02 Mar 2015 - </p> - - <p><p>February might be the shortest month of the year, but this does not -mean that the Flink community has not been busy adding features to the -system and fixing bugs. Here’s a rundown of the activity in the Flink -community last month.</p> - -</p> - - <p><a href="/news/2015/03/02/february-2015-in-flink.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -350,6 +346,16 @@ community last month.</p> <ul id="markdown-toc"> + <li><a href="/2019/05/03/pulsar-flink.html">When Flink & Pulsar Come Together</a></li> + + + + + + + + + <li><a href="/news/2019/04/17/sod.html">Apache Flink's Application to Season of Docs</a></li> diff --git a/content/blog/page8/index.html b/content/blog/page8/index.html index b6a385c..9e45979 100644 --- a/content/blog/page8/index.html +++ b/content/blog/page8/index.html @@ -159,6 +159,24 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2015/03/02/february-2015-in-flink.html">February 2015 in the Flink community</a></h2> + + <p>02 Mar 2015 + </p> + + <p><p>February might be the shortest month of the year, but this does not +mean that the Flink community has not been busy adding features to the +system and fixing bugs. Here’s a rundown of the activity in the Flink +community last month.</p> + +</p> + + <p><a href="/news/2015/03/02/february-2015-in-flink.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2015/02/09/streaming-example.html">Introducing Flink Streaming</a></h2> <p>09 Feb 2015 @@ -332,6 +350,16 @@ academic and open source project that Flink originates from.</p> <ul id="markdown-toc"> + <li><a href="/2019/05/03/pulsar-flink.html">When Flink & Pulsar Come Together</a></li> + + + + + + + + + <li><a href="/news/2019/04/17/sod.html">Apache Flink's Application to Season of Docs</a></li> diff --git a/content/img/blog/pulsar-flink/image-1.png b/content/img/blog/pulsar-flink/image-1.png new file mode 100644 index 0000000..8d3a9a1 Binary files /dev/null and b/content/img/blog/pulsar-flink/image-1.png differ diff --git a/content/img/blog/pulsar-flink/image-2.png b/content/img/blog/pulsar-flink/image-2.png new file mode 100644 index 0000000..b046402 Binary files /dev/null and b/content/img/blog/pulsar-flink/image-2.png differ diff --git a/content/img/blog/pulsar-flink/image-3.png b/content/img/blog/pulsar-flink/image-3.png new file mode 100644 index 0000000..6c85861 Binary files /dev/null and b/content/img/blog/pulsar-flink/image-3.png differ diff --git a/content/img/blog/pulsar-flink/image-4.png b/content/img/blog/pulsar-flink/image-4.png new file mode 100644 index 0000000..7363852 Binary files /dev/null and b/content/img/blog/pulsar-flink/image-4.png differ diff --git a/content/img/blog/pulsar-flink/image-5.png b/content/img/blog/pulsar-flink/image-5.png new file mode 100644 index 0000000..188caf4 Binary files /dev/null and b/content/img/blog/pulsar-flink/image-5.png differ diff --git a/content/index.html b/content/index.html index e538d3c..601e099 100644 --- a/content/index.html +++ b/content/index.html @@ -448,6 +448,9 @@ <dl> + <dt> <a href="/2019/05/03/pulsar-flink.html">When Flink & Pulsar Come Together</a></dt> + <dd>Apache Flink and Apache Pulsar are distributed data processing systems. When combined, they offer elastic data processing at large scale. This post describes how Pulsar and Flink can work together to provide a seamless developer experience.</dd> + <dt> <a href="/news/2019/04/17/sod.html">Apache Flink's Application to Season of Docs</a></dt> <dd><p>The Apache Flink community is happy to announce its application to the first edition of <a href="https://developers.google.com/season-of-docs/";>Season of Docs</a> by Google. The program is bringing together Open Source projects and technical writers to raise awareness for and improve documentation of Open Source projects. While the community is continuously looking for new contributors to collaborate on our documentation, we would like to take this chance to work with one [...] @@ -467,9 +470,6 @@ for more details.</p> <dt> <a href="/news/2019/03/06/ffsf-preview.html">What to expect from Flink Forward San Francisco 2019</a></dt> <dd>The third annual Flink Forward conference in San Francisco is just a few weeks away. Let's see what Flink Forward SF 2019 has in store for the Apache Flink and stream processing communities. This post covers some of its highlights!</dd> - - <dt> <a href="/news/2019/02/25/monitoring-best-practices.html">Monitoring Apache Flink Applications 101</a></dt> - <dd>The monitoring of business-critical applications is a crucial aspect of a production deployment. It ensures that any degradation or downtime is immediately identified and can be resolved as quickly as possible. In this post, we discuss the most important metrics that indicate healthy Flink applications.</dd> </dl> diff --git a/content/zh/index.html b/content/zh/index.html index da08060..8c98651 100644 --- a/content/zh/index.html +++ b/content/zh/index.html @@ -446,6 +446,9 @@ <dl> + <dt> <a href="/2019/05/03/pulsar-flink.html">When Flink & Pulsar Come Together</a></dt> + <dd>Apache Flink and Apache Pulsar are distributed data processing systems. When combined, they offer elastic data processing at large scale. This post describes how Pulsar and Flink can work together to provide a seamless developer experience.</dd> + <dt> <a href="/news/2019/04/17/sod.html">Apache Flink's Application to Season of Docs</a></dt> <dd><p>The Apache Flink community is happy to announce its application to the first edition of <a href="https://developers.google.com/season-of-docs/";>Season of Docs</a> by Google. The program is bringing together Open Source projects and technical writers to raise awareness for and improve documentation of Open Source projects. While the community is continuously looking for new contributors to collaborate on our documentation, we would like to take this chance to work with one [...] @@ -465,9 +468,6 @@ for more details.</p> <dt> <a href="/news/2019/03/06/ffsf-preview.html">What to expect from Flink Forward San Francisco 2019</a></dt> <dd>The third annual Flink Forward conference in San Francisco is just a few weeks away. Let's see what Flink Forward SF 2019 has in store for the Apache Flink and stream processing communities. This post covers some of its highlights!</dd> - - <dt> <a href="/news/2019/02/25/monitoring-best-practices.html">Monitoring Apache Flink Applications 101</a></dt> - <dd>The monitoring of business-critical applications is a crucial aspect of a production deployment. It ensures that any degradation or downtime is immediately identified and can be resolved as quickly as possible. In this post, we discuss the most important metrics that indicate healthy Flink applications.</dd> </dl>
