This is an automated email from the ASF dual-hosted git repository. pnowojski pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit fffa4425a4b328bf121925a1de62134254256182 Author: Piotr Nowojski <[email protected]> AuthorDate: Tue Jul 7 15:52:08 2020 +0200 Rebuild website --- content/blog/feed.xml | 808 ++++++++++++++++++----------------------- content/blog/index.html | 36 +- content/blog/page10/index.html | 43 ++- content/blog/page11/index.html | 43 ++- content/blog/page12/index.html | 25 ++ content/blog/page2/index.html | 38 +- content/blog/page3/index.html | 41 ++- content/blog/page4/index.html | 39 +- content/blog/page5/index.html | 38 +- content/blog/page6/index.html | 40 +- content/blog/page7/index.html | 40 +- content/blog/page8/index.html | 40 +- content/blog/page9/index.html | 40 +- content/index.html | 8 +- content/zh/index.html | 8 +- 15 files changed, 655 insertions(+), 632 deletions(-) diff --git a/content/blog/feed.xml b/content/blog/feed.xml index d0c54b9..2f8e703 100644 --- a/content/blog/feed.xml +++ b/content/blog/feed.xml @@ -7,6 +7,357 @@ <atom:link href="https://flink.apache.org/blog/feed.xml"; rel="self" type="application/rss+xml" /> <item> +<title>Apache Flink 1.11.0 Release Announcement</title> +<description><p>The Apache Flink community is proud to announce the release of Flink 1.11.0! More than 200 contributors worked on over 1.3k issues to bring significant improvements to usability as well as new features to Flink users across the whole API stack. Some highlights that we’re particularly excited about are:</p> + +<ul> + <li> + <p>The core engine is introducing <strong>unaligned checkpoints</strong>, a major change to Flink’s fault tolerance mechanism that improves checkpointing performance under heavy backpressure.</p> + </li> + <li> + <p>A <strong>new Source API</strong> that simplifies the implementation of (custom) sources by unifying batch and streaming execution, as well as offloading internals such as event-time handling, watermark generation or idleness detection to Flink.</p> + </li> + <li> + <p>Flink SQL is introducing <strong>Support for Change Data Capture (CDC)</strong> to easily consume and interpret database changelogs from tools like Debezium. The renewed <strong>FileSystem Connector</strong> also expands the set of use cases and formats supported in the Table API/SQL, enabling scenarios like streaming data directly from Kafka to Hive.</p> + </li> + <li> + <p>Multiple performance optimizations to PyFlink, including support for <strong>vectorized User-defined Functions (Pandas UDFs)</strong>. This improves interoperability with libraries like Pandas and NumPy, making Flink more powerful for data science and ML workloads.</p> + </li> +</ul> + +<p>Read on for all major new features and improvements, important changes to be aware of and what to expect moving forward!</p> + +<div class="page-toc"> +<ul id="markdown-toc"> + <li><a href="#new-features-and-improvements" id="markdown-toc-new-features-and-improvements">New Features and Improvements</a> <ul> + <li><a href="#unaligned-checkpoints-beta" id="markdown-toc-unaligned-checkpoints-beta">Unaligned Checkpoints (Beta)</a></li> + <li><a href="#unified-watermark-generators" id="markdown-toc-unified-watermark-generators">Unified Watermark Generators</a></li> + <li><a href="#new-data-source-api-beta" id="markdown-toc-new-data-source-api-beta">New Data Source API (Beta)</a></li> + <li><a href="#application-mode-deployments" id="markdown-toc-application-mode-deployments">Application Mode Deployments</a></li> + <li><a href="#other-improvements" id="markdown-toc-other-improvements">Other Improvements</a></li> + <li><a href="#table-apisql-support-for-change-data-capture-cdc" id="markdown-toc-table-apisql-support-for-change-data-capture-cdc">Table API/SQL: Support for Change Data Capture (CDC)</a></li> + <li><a href="#table-apisql-jdbc-catalog-interface-and-postgres-catalog" id="markdown-toc-table-apisql-jdbc-catalog-interface-and-postgres-catalog">Table API/SQL: JDBC Catalog Interface and Postgres Catalog</a></li> + <li><a href="#table-apisql-filesystem-connector-with-support-for-avro-orc-and-parquet" id="markdown-toc-table-apisql-filesystem-connector-with-support-for-avro-orc-and-parquet">Table API/SQL: FileSystem Connector with Support for Avro, ORC and Parquet</a></li> + <li><a href="#table-apisql-support-for-python-udfs" id="markdown-toc-table-apisql-support-for-python-udfs">Table API/SQL: Support for Python UDFs</a></li> + <li><a href="#other-improvements-to-the-table-apisql" id="markdown-toc-other-improvements-to-the-table-apisql">Other Improvements to the Table API/SQL</a></li> + <li><a href="#pyflink-support-for-pandas-udfs" id="markdown-toc-pyflink-support-for-pandas-udfs">PyFlink: Support for Pandas UDFs</a></li> + <li><a href="#other-improvements-to-pyflink" id="markdown-toc-other-improvements-to-pyflink">Other Improvements to PyFlink</a></li> + </ul> + </li> + <li><a href="#important-changes" id="markdown-toc-important-changes">Important Changes</a></li> + <li><a href="#release-notes" id="markdown-toc-release-notes">Release Notes</a></li> + <li><a href="#list-of-contributors" id="markdown-toc-list-of-contributors">List of Contributors</a></li> +</ul> + +</div> + +<p>The binary distribution and source artifacts are now available on the updated <a href="/downloads.html">Downloads page</a> of the Flink website, and the most recent distribution of PyFlink is available on <a href="https://pypi.org/project/apache-flink/">PyPI</a>. Please review the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html">release notes</a> carefully, and c [...] + +<p>We encourage you to download the release and share your feedback with the community through the <a href="https://flink.apache.org/community.html#mailing-lists">Flink mailing lists</a> or <a href="https://issues.apache.org/jira/projects/FLINK/summary">JIRA</a>.</p> + +<h2 id="new-features-and-improvements">New Features and Improvements</h2> + +<h3 id="unaligned-checkpoints-beta">Unaligned Checkpoints (Beta)</h3> + +<p>Triggering a checkpoint in Flink will cause a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/internals/stream_checkpointing.html#barriers">checkpoint barrier</a> to flow from the sources of your topology all the way towards the sinks. For operators that receive more than one input stream, the barriers flowing through each channel need to be aligned before the operator can snapshot its state and forward the checkpoint barrier — typical [...] + +<ul> + <li> + <p>Checkpoint barriers will flow much slower through backpressured channels, effectively blocking the remaining channels and their upstream operators during checkpointing;</p> + </li> + <li> + <p>Slow checkpoint barrier propagation leads to longer checkpointing times and can, worst case, result in little to no progress in the application.</p> + </li> +</ul> + +<p>To improve the performance of checkpointing under backpressure scenarios, the community is rolling out the first iteration of unaligned checkpoints (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints">FLIP-76</a>) with Flink 1.11. Compared to the original checkpointing mechanism (Fig. 1), this approach doesn’t wait for barrier alignment across input channels, instead allowing barriers to overtake in-flight records (i [...] + +<div style="line-height:60%;"> + <br /> +</div> + +<div class="row"> + <div class="col-lg-6"> + <div class="text-center"> + <figure> + <img src="/img/blog/2020-07-06-release-1.11.0/image1.gif" width="600px" alt="Aligned Checkpoints" /> + <br /><br /> + <figcaption><i><b>Fig.1:</b> Aligned Checkpoints</i></figcaption> + </figure> + </div> + </div> + <div class="col-lg-6"> + <div class="text-center"> + <figure> + <img src="/img/blog/2020-07-06-release-1.11.0/image2.png" width="600px" alt="Unaligned Checkpoints" /> + <br /><br /> + <figcaption><i><b>Fig.2:</b> Unaligned Checkpoints</i></figcaption> + </figure> + </div> + </div> +</div> + +<div style="line-height:150%;"> + <br /> +</div> + +<p>Because in-flight records have to be persisted as part of the snapshot, unaligned checkpoints will lead to increased checkpoints sizes. On the upside, <strong>checkpointing times are heavily reduced</strong>, so users will see more progress (even in unstable environments) as more up-to-date checkpoints will lighten the recovery process. You can learn more about the current limitations of unaligned checkpoints in the <a href="https://ci.apache.org/projects/fli [...] + +<p>As with any beta feature, we appreciate early feedback that you might want to share with the community after giving unaligned checkpoints a try!</p> + +<p><span class="label label-info">Info</span> To enable this feature, you need to configure the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/streaming/api/environment/CheckpointConfig.html"><code>enableUnalignedCheckpoints</code></a> option in your <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/stream/state/checkpointing.html#enabling-and-con [...] + +<h3 id="unified-watermark-generators">Unified Watermark Generators</h3> + +<p>So far, watermark generation (prev. also called <em>assignment</em>) relied on two different interfaces: <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/streaming/api/functions/AssignerWithPunctuatedWatermarks.html"><code>AssignerWithPunctuatedWatermarks</code></a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/streaming/api/f [...] + +<p>This gives users more control over watermark emission and simplifies the implementation of new connectors that need to support watermark assignment and timestamp extraction at the source (see <em><a href="#new-data-source-api-beta">New Data Source API</a></em>). Multiple <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11//dev/event_timestamps_watermarks.html#introduction-to-watermark-strategies">strategies for [...] + +<p><strong>Support for Watermark Idleness Detection</strong></p> + +<p>The <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/api/common/eventtime/WatermarkStrategy.html#withIdleness-java.time.Duration-"><code>WatermarkStrategy.withIdleness()</code></a> method allows you to mark a stream as idle if no events arrive within a configured time (i.e. a timeout duration), which in turn allows handling event time skew properly and preventing idle partitions from holding back th [...] + +<p><span class="label label-info">Note</span> <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-126%3A+Unify+%28and+separate%29+Watermark+Assigners">FLIP-126</a> introduces no breaking changes, but we recommend that users give preference to the new <code>WatermarkGenerator</code> interface moving forward, in preparation for the deprecation of the legacy watermark assigners in future releases.</p> + +<h3 id="new-data-source-api-beta">New Data Source API (Beta)</h3> + +<p>Up to this point, writing a production-grade source connector for Flink was a non-trivial task that required users to be somewhat familiar with Flink internals and account for implementation details like event time assignment, watermark generation or idleness detection in their code. Flink 1.11 introduces a new Data Source API (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface">FLIP-27</a>) to overcome these lim [...] + +<center> + <figure> + <img src="/img/blog/2020-07-06-release-1.11.0/image3.png" width="600px" alt="Data Source API" /> + </figure> +</center> + +<div style="line-height:150%;"> + <br /> +</div> + +<p>Separating the work of split discovery and the actual reading of the consumed data (i.e. the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/stream/sources.html#data-source-concepts"><em>splits</em></a>) in different components — resp. the <code>SplitEnumerator</code> and <code>SourceReader</code> — allows mixing and matching different enumeration strategies and split readers.</p> + +<p>As an example, the existing Kafka connector has multiple strategies for partition discovery that are intermingled with the rest of the code. With the new interfaces in place, it would only need a single reader implementation and there could be several split enumerators for the different partition discovery strategies.</p> + +<p><strong>Batch and Streaming Unification</strong></p> + +<p>Source connectors implemented using the Data Source API will be able to work both as a bounded (<em>batch</em>) and unbounded (<em>streaming</em>) source. The difference between both cases is minimal: for bounded input, the <code>SplitEnumerator</code> will generate a fixed set of splits and each split is finite; for unbounded input, either the splits are not finite or the <code>SplitEnumerator</code> keeps generating new splits.&l [...] + +<p><strong>Implicit Watermark and Event Time Handling</strong></p> + +<p>The <code>TimestampAssigner</code> and <code>WatermarkGenerator</code> run transparently as part of the <code>SourceReader</code> component, so users also don’t have to implement any timestamp extraction or watermark generation code.</p> + +<p><span class="label label-info">Note</span> The existing source connectors have not yet been reimplemented using the Data Source API — this is planned for upcoming releases. If you’re looking to implement a new source, please refer to the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/stream/sources.html#data-sources">Data Source documentation</a> and <a href="https://ci.apache.org/projects/flink/flink [...] + +<h3 id="application-mode-deployments">Application Mode Deployments</h3> + +<p>Prior to Flink 1.11, jobs in a Flink application could either be submitted to a long-running <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/concepts/flink-architecture.html#flink-session-cluster">Flink Session Cluster</a> (<em>session mode</em>) or a dedicated <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/concepts/flink-architecture.html#flink-job-cluster">Flink Job Cluster</a> [...] + +<p>From this release on, Flink gets an additional deployment mode: <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/#application-mode">Application Mode</a> (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode">FLIP-85</a>); where the <code>main()</code> method runs on the cluster, rather than the client. The job submission becomes a one-step process: you [...] + +<p>In Flink 1.11, the community worked to already support <em>application mode</em> in Kubernetes (<a href="https://issues.apache.org/jira/browse/FLINK-10934">FLINK-10934</a>).</p> + +<h3 id="other-improvements">Other Improvements</h3> + +<p><strong>Unified Memory Configuration for JobManagers (<a href="https://jira.apache.org/jira/browse/FLINK-16614">FLIP-116</a>)</strong></p> + +<p>Following the work started in Flink 1.10 to improve memory management and configuration, this release introduces a new memory model that aligns the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_master.html">JobManagers’ configuration options</a> and terminology with that introduced in <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors" [...] + +<p><span class="label label-danger">Attention</span> Reusing a previous Flink configuration without any adjustments can result in differently computed memory parameters for the JVM and, as a result, performance changes or even failures. Make sure to check the <a href="https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html#migrate-job-manager-memory-configuration">migration guide</a> if you’re planning to upda [...] + +<p><strong>Improvements to the Flink WebUI (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-75%3A+Flink+Web+UI+Improvement+Proposal">FLIP-75</a>)</strong></p> + +<p>In Flink 1.11, the community kicked off a series of improvements to the Flink WebUI. The first to roll out are better TaskManager and JobManager log display (<a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143">FLIP-103</a>), as well as a new thread dump utility (<a href="https://issues.apache.org/jira/browse/FLINK-14816">FLINK-14816</a>). Some additional work planned for upcoming releases includes bette [...] + +<p><strong>Docker Image Unification (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification">FLIP-111</a>)</strong></p> + +<p>With this release, all Docker-related resources have been consolidated into <a href="https://github.com/apache/flink-docker">apache/flink-docker</a> and the entry point script has been extended to allow users to run the default Docker image in different modes without the need to create a custom image. The <a href="https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/docker.html#customize-flink-image">updated documentation< [...] + +<hr /> + +<h3 id="table-apisql-support-for-change-data-capture-cdc">Table API/SQL: Support for Change Data Capture (CDC)</h3> + +<p>Change Data Capture (CDC) has become a popular pattern to capture committed changes from a database and propagate those changes to downstream consumers, for example to keep multiple datastores in sync and avoid common pitfalls such as <a href="https://thorben-janssen.com/dual-writes/">dual writes</a>. Being able to easily ingest and interpret these changelogs into the Table API/SQL has been a highly demanded feature in the Flink community — and it’s now pos [...] + +<p>To extend the scope of the Table API/SQL to use cases like CDC, Flink 1.11 introduces new table source and sink interfaces with <strong>changelog mode</strong> (see <em><a href="#other-improvements-to-the-table-apisql">New TableSource and TableSink Interfaces</a></em>) and support for the <a href="https://debezium.io/">Debezium</a> and <a href="https://github.com/alibaba/canal">Canal</a> fo [...] + +<center> + <figure> + <img src="/img/blog/2020-07-06-release-1.11.0/image4.png" width="500px" alt="CDC" /> + </figure> +</center> + +<div style="line-height:150%;"> + <br /> +</div> + +<p>Users have to specify either <code>“format=debezium-json”</code> or <code>“format=canal-json”</code> in their <code>CREATE TABLE</code> statement to consume changelogs using SQL DDL.</p> + +<div class="highlight"><pre><code class="language-sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_table</span> <span class="p">(</span> + <span class="p">...</span> +<span class="p">)</span> <span class="k">WITH</span> <span class="p">(</span> + <span class="s1">&#39;connector&#39;</span><span class="o">=</span><span class="s1">&#39;...&#39;</span><span class="p">,</span> <span class="c1">-- e.g. &#39;kafka&#39;</span> + <span class="s1">&#39;format&#39;</span><span class="o">=</span><span class="s1">&#39;debezium-json&#39;</span><span class="p">,</span> + <span class="s1">&#39;debezium-json.schema-include&#39;</span><span class="o">=</span><span class="s1">&#39;true&#39;</span> <span class="c1">-- default: false (Debezium can be configured to include or exclude the message schema)</span> + <span class="s1">&#39;debezium-json.ignore-parse-errors&#39;</span><span class="o">=</span><span class="s1">&#39;true&#39;</span> <span class="c1">-- default: false</span> +<span class="p">);</span></code></pre></div> + +<p>Flink 1.11 only supports Kafka as a changelog source out-of-the-box and JSON-encoded changelogs, with Avro (Debezium) and Protobuf (Canal) planned for future releases. There are also plans to support MySQL binlogs and Kafka compacted topics as sources, as well as to extend changelog support to batch execution.</p> + +<p><span class="label label-danger">Attention</span> There is a known issue (<a href="https://issues.apache.org/jira/browse/FLINK-18461">FLINK-18461</a>) that prevents changelog sources from being used to write to upsert sinks (e.g. MySQL, HBase, Elasticsearch). This will be fixed in the next patch release (1.11.1).</p> + +<h3 id="table-apisql-jdbc-catalog-interface-and-postgres-catalog">Table API/SQL: JDBC Catalog Interface and Postgres Catalog</h3> + +<p>Flink 1.11 introduces a generic JDBC catalog interface (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-93%3A+JDBC+catalog+and+Postgres+catalog">FLIP-93</a>) that enables users of the Table API/SQL to <strong>derive table schemas automatically</strong> from connections to relational databases over <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connect.html#jdbc-connector">JDBC</ [...] + +<p>The first implementation, rolling out with the new release, is the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/catalogs.html#postgrescatalog">Postgres catalog</a>.</p> + +<h3 id="table-apisql-filesystem-connector-with-support-for-avro-orc-and-parquet">Table API/SQL: FileSystem Connector with Support for Avro, ORC and Parquet</h3> + +<p>To improve the user experience for end-to-end streaming ETL use cases, the Flink community worked on a new FileSystem Connector for the Table API/SQL (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-115%3A+Filesystem+connector+in+Table">FLIP-115</a>). The implementation is based on Flink’s <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/filesystems/index.html">FileSystem abstraction</a> and reu [...] + +<p>This also means that Table API/SQL users can now make use of all formats already supported by StreamingFileSink, like (Avro) Parquet, as well as the new formats introduced with this release, like Avro (<a href="https://issues.apache.org/jira/browse/FLINK-11395">FLINK-11395</a>) and Orc (<a href="https://issues.apache.org/jira/browse/FLINK-10114">FLINK-10114</a>).</p> + +<div class="highlight"><pre><code class="language-sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_table</span> <span class="p">(</span> + <span class="n">column_name1</span> <span class="nb">INT</span><span class="p">,</span> + <span class="n">column_name2</span> <span class="n">STRING</span><span class="p">,</span> + <span class="p">...</span> + <span class="n">part_name1</span> <span class="nb">INT</span><span class="p">,</span> + <span class="n">part_name2</span> <span class="n">STRING</span> +<span class="p">)</span> <span class="n">PARTITIONED</span> <span class="k">BY</span> <span class="p">(</span><span class="n">part_name1</span><span class="p">,</span> <span class="n">part_name2</span><span class="p">)</span> <span class="k">WITH</span> <span class="p">(</span> + <span class="s1">&#39;connector&#39;</span> <span class="o">=</span> <span class="s1">&#39;filesystem&#39;</span><span class="p">,</span> + <span class="s1">&#39;path&#39;</span> <span class="o">=</span> <span class="s1">&#39;file:///path/to/file,</span> +<span class="s1"> &#39;</span><span class="n">format</span><span class="s1">&#39; = &#39;</span><span class="p">...</span><span class="err">&#39;</span><span class="p">,</span> <span class="c1">-- supported formats: Avro, ORC, Parquet, CSV, JSON </span> + <span class="p">...</span> +<span class="p">);</span></code></pre></div> + +<p>The new all-rounder FileSystem Connector transparently handles batch and streaming execution, provides exactly-once guarantees and has full partition support, greatly expanding the scope of usage of the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connect.html#file-system-connector">legacy connector</a>. This allows users to easily implement common use cases like <strong>directly streaming data from Kafka to Hive< [...] + +<p>You can track the upcoming improvements to the FileSystem Connector in <a href="https://issues.apache.org/jira/browse/FLINK-17778">FLINK-17778</a>.</p> + +<h3 id="table-apisql-support-for-python-udfs">Table API/SQL: Support for Python UDFs</h3> + +<p>Prior to this release, users of the Table API/SQL were limited to defining UDFs in either Java or Scala. In Flink 1.11, the community worked on expanding the usage scope of the Python language beyond PyFlink and providing support for Python UDFs in the SQL DDL syntax (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL">FLIP-106</a>), as well as the SQL Client (<a href="https://cwiki.apache.o [...] + +<h3 id="other-improvements-to-the-table-apisql">Other Improvements to the Table API/SQL</h3> + +<p><strong>DDL and DML Compatibility for the Hive Connector (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-123%3A+DDL+and+DML+compatibility+for+Hive+connector">FLIP-123</a>)</strong></p> + +<p>Starting from Flink 1.11, users can write SQL statements directly using Hive syntax (HiveQL) in the Table API/SQL and the SQL Client. For this purpose, an additional dialect was introduced and users can now dynamically switch between Flink (<code>default</code>) and Hive (<code>hive</code>) on a per-statement basis. For a complete list of supported DDL and DML statements, check the Hive dialect <a href="https://ci.apache.org/projects/flink/flink-d [...] + +<p><strong>Extensions and Improvements to the Flink SQL Syntax</strong></p> + +<ul> + <li> + <p>Flink 1.11 introduces the concept of <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/sql/create.html#create-table">primary key constraints</a> to leverage runtime optimizations in Flink SQL DDL (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP+87%3A+Primary+key+constraints+in+Table+API">FLIP-87</a>);</p> + </li> + <li> + <p>View objects are now fully supported in SQL DDL using the <code>CREATE</code>/<code>ALTER</code>/<code>DROP VIEW</code> statements (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-71%3A+E2E+View+support+in+FLINK+SQL">FLIP-71</a>);</p> + </li> + <li> + <p>Users can now specify or override table options in their DQL/DML statements using <a href="https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/hints.html#dynamic-table-options">dynamic table options</a> (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL">FLIP-113</a>).</p> + </li> + <li> + <p>To make connector properties less verbose and improve exception handling, some key properties have been refactored (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-122%3A+New+Connector+Property+Keys+for+New+Factory">FLIP-122</a>). This change does not break compatibility, so users can still use the old property keys.</p> + </li> +</ul> + +<p><strong>New TableSource and TableSink Interfaces (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces">FLIP-95</a>)</strong></p> + +<p>Flink 1.11 introduces new table source and sink interfaces (resp. <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/table/connector/source/DynamicTableSource.html"><code>DynamicTableSource</code></a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/table/connector/sink/DynamicTableSink.html"><code>DynamicTableSink</code>&l [...] + +<p><span class="label label-info">Note</span> Although compatibility is not immediately affected, we recommend that Table API/SQL users update any sources and sinks to the new interface stack.</p> + +<p><strong>Refactored TableEnvironment Interface (<a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=134745878">FLIP-84</a>)</strong></p> + +<p>The semantics to describe similar behaviours in the <code>TableEnvironment</code> and <code>Table</code> interfaces have diverged over time, leading to an inconsistent and sometimes unclear user experience. To improve this and make programming more fluent in the Table API/SQL, Flink 1.11 introduces new methods that unify behaviours like execution triggering (e.g. <code>executeSql()</code>) and result representation (e.g. <a href="https [...] + +<p><span class="label label-info">Note</span> The methods deprecated with FLIP-84 will not be immediately removed, but we recommend that users adopt the newly introduced methods. For a complete list of new and deprecated methods, check the “Summary” section of <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=134745878">FLIP-84</a>.</p> + +<p><strong>New Type Inference for Table API UDFs (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-65%3A+New+type+inference+for+Table+API+UDFs">FLIP-65</a>)</strong></p> + +<p>In Flink 1.9, the community started working on a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/types.html#data-types">new data type system</a> for the Table API to improve its compliance with the SQL standard (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System">FLIP-37</a>). This work is now close to being completed in Flink 1.11, with the exposure [...] + +<hr /> + +<h3 id="pyflink-support-for-pandas-udfs">PyFlink: Support for Pandas UDFs</h3> + +<p>Up to this release, Python UDFs in PyFlink only supported scalar values of standard Python types. This presented some limitations:</p> + +<ul> + <li> + <p>High serialization/deserialization overhead in the process of transferring data between the JVM and the Python processes;</p> + </li> + <li> + <p>Hard to integrate with common Python libraries for high-performance numerical processing like pandas and NumPy.</p> + </li> +</ul> + +<p>To overcome these limitations, the community introduced support for (scalar) <strong>vectorized Python UDFs</strong> based on <a href="https://pandas.pydata.org/pandas-docs/stable/getting_started/overview.html">pandas</a> in Flink 1.11 (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-97%3A+Support+Scalar+Vectorized+Python+UDF+in+PyFlink">FLIP-97</a>). The performance of vectorized UDFs is usually much higher [...] + +<div class="highlight"><pre><code class="language-python"><span class="nd">@udf</span><span class="p">(</span><span class="n">input_types</span><span class="o">=</span><span class="p">[</span><span class="n">DataTypes</span><span class="o">.</span><span class="n">BIGINT</span>& [...] +<span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">):</span> + <span class="k">return</span> <span class="n">i</span> <span class="o">+</span> <span class="n">j</span></code></pre></div> + +<p>To mark a UDF as a Pandas UDF, you only need to add an extra parameter <code>udf_type=”pandas”</code> in the udf decorator, as described in the <a href="https://ci.apache.org/projects/flink/flink-docs-master/dev/table/python/vectorized_python_udfs.html#vectorized-user-defined-functions">documentation</a>.</p> + +<h3 id="other-improvements-to-pyflink">Other Improvements to PyFlink</h3> + +<p><strong>Conversion fromPandas/toPandas (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-120%3A+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame">FLIP-120</a>)</strong></p> + +<p>Arrow is also supported as an optimization to convert between PyFlink tables and <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html"><code>pandas.DataFrames</code></a>, enabling users to switch processing engines seamlessly without the need for an intermediate connector. For examples on how to use the new <code>fromPandas()</code> and <code>toPandas()</code> methods in PyFlink, check [...] + +<p><strong>Support for User-defined Table Functions (UDTFs) (<a href="https://jira.apache.org/jira/browse/FLINK-14500">FLINK-14500</a>)</strong></p> + +<p>From Flink 1.11, you can define and register custom <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/python/python_udfs.html#table-functions">UDTFs</a> in PyFlink. Similar to a Python UDF, a UDTF takes zero, one or multiple scalar values as input, but can return an arbitrary number of rows as output instead of a single value.</p> + +<p><strong>Cython Performance Optimization for UDFs (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-121%3A+Support+Cython+Optimizing+Python+User+Defined+Function">FLIP-121</a>)</strong></p> + +<p><a href="https://cython.readthedocs.io/en/latest/src/quickstart/cythonize.html">Cython</a> is a compiled superset of the Python language that is often used to improve the performance of large-scale numeric processing in Python, as it optimizes execution to machine code-level speed and pairs well with popular C-based libraries like NumPy. From Flink 1.11, you can build <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/flinkDev/bui [...] + +<p><strong>User-defined Metrics in Python UDFs (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-112%3A+Support+User-Defined+Metrics+in++Python+UDF">FLIP-112</a>)</strong></p> + +<p>To make it easier for users to monitor and debug the execution of Python UDFs, PyFlink now allows gathering and exposing metrics to external systems, as well as defining user scopes and variables. You can access the metrics system from a UDF by calling <code>function_context.get_metric_group()</code> in the open method, as described in the <a href="https://ci.apache.org/projects/flink/flink-docs-master/dev/table/python/metrics.html#registering-metrics"&g [...] + +<hr /> + +<h2 id="important-changes">Important Changes</h2> + +<ul> + <li> + <p>[<a href="https://jira.apache.org/jira/browse/FLINK-17339">FLINK-17339</a>] The Blink planner is the <strong>default</strong> in the Table API/SQL starting from Flink 1.11. This was already the case for the SQL Client since Flink 1.10. The old Flink planner is still supported, but not actively developed.</p> + </li> + <li> + <p>[<a href="https://issues.apache.org/jira/browse/FLINK-5763">FLINK-5763</a>] Savepoints now contain all their state inside a single directory (both metadata and program state). This makes it straightforward to figure out which files make up the state of a savepoint and allows users to <strong>relocate savepoints</strong> by simply moving a directory.</p> + </li> + <li> + <p>[<a href="https://issues.apache.org/jira/browse/FLINK-16408">FLINK-16408</a>] To reduce pressure on the JVM metaspace, the user code class loader is being reused by a <code>TaskExecutor</code> as long as there is at least a single slot allocated for the respective job. This changes Flink’s recovery behaviour slightly, so that it will not reload static fields.</p> + </li> + <li> + <p>[<a href="https://issues.apache.org/jira/browse/FLINK-11086">FLINK-11086</a>] Flink now supports Hadoop versions above <strong>Hadoop 3.0.0</strong>. Note that the Flink project does not provide any updated “flink-shaded-hadoop-*” jars. Users need to provide Hadoop dependencies through the <code>HADOOP_CLASSPATH</code> environment variable (recommended) or the lib/ folder.</p> + </li> + <li> + <p>[<a href="https://issues.apache.org/jira/browse/FLINK-16963">FLINK-16963</a>] All <code>MetricReporters</code> that come with Flink have been converted to plugins. These should no longer be placed into <code>/lib</code> (which may result in dependency conflicts), but <code>/plugins/&lt;some_directory&gt;</code> instead.</p> + </li> + <li> + <p>[<a href="https://issues.apache.org/jira/browse/FLINK-12639">FLINK-12639</a>] The Flink <strong>documentation</strong> is undergoing some <strong>rework</strong>, so you might notice that the navigation and organization of content look slightly different starting from Flink 1.11.</p> + </li> +</ul> + +<h2 id="release-notes">Release Notes</h2> + +<p>Please review the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html">release notes</a> carefully for a detailed list of changes and new features if you plan to upgrade your setup to Flink 1.11. This version is API-compatible with previous 1.x releases for APIs annotated with the @Public annotation.</p> + +<h2 id="list-of-contributors">List of Contributors</h2> + +<p>The Apache Flink community would like to thank all the 200+ contributors that have made this release possible:</p> + +<p>Aitozi, Alexander Fedulov, Alexey Trenikhin, Aljoscha Krettek, Andrey Zagrebin, Arvid Heise, Ayush Saxena, Bairos, Bartosz Krasinski, Benchao Li, Benoit Hanotte, Benoît Paris, Bhagavan Das, Canbin Zheng, Cedric Chen, Chesnay Schepler, Colm O hEigeartaigh, Congxian Qiu, CrazyTomatoOo, Danish Amjad, Danny Chan, David Anderson, Dawid Wysakowicz, Dian Fu, Dominik Wosiński, Echo Lee, Ethan Marsh, Etienne Chauchot, Fabian Hueske, Fabian Paul, Flavio Pompermaier, Gao Yun, Gary Yao, Ghi [...] +</description> +<pubDate>Mon, 06 Jul 2020 10:00:00 +0200</pubDate> +<link>https://flink.apache.org/news/2020/07/06/release-1.11.0.html</link> +<guid isPermaLink="true">/news/2020/07/06/release-1.11.0.html</guid> +</item> + +<item> <title>Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 2</title> <description><p>In a previous post, we introduced the basics of Flink on Zeppelin and how to do Streaming ETL. In this second part of the “Flink on Zeppelin” series of posts, I will share how to perform streaming data visualization via Flink on Zeppelin and how to use Apache Flink UDFs in Zeppelin.</p> @@ -16426,462 +16777,5 @@ for this release:</p> <guid isPermaLink="true">/news/2015/09/01/release-0.9.1.html</guid> </item> -<item> -<title>Introducing Gelly: Graph Processing with Apache Flink</title> -<description><p>This blog post introduces <strong>Gelly</strong>, Apache Flink’s <em>graph-processing API and library</em>. Flink’s native support -for iterations makes it a suitable platform for large-scale graph analytics. -By leveraging delta iterations, Gelly is able to map various graph processing models such as -vertex-centric or gather-sum-apply to Flink dataflows.</p> - -<p>Gelly allows Flink users to perform end-to-end data analysis in a single system. -Gelly can be seamlessly used with Flink’s DataSet API, -which means that pre-processing, graph creation, analysis, and post-processing can be done -in the same application. At the end of this post, we will go through a step-by-step example -in order to demonstrate that loading, transformation, filtering, graph creation, and analysis -can be performed in a single Flink program.</p> - -<p><strong>Overview</strong></p> - -<ol> - <li><a href="#what-is-gelly">What is Gelly?</a></li> - <li><a href="#graph-representation-and-creation">Graph Representation and Creation</a></li> - <li><a href="#transformations-and-utilities">Transformations and Utilities</a></li> - <li><a href="#iterative-graph-processing">Iterative Graph Processing</a></li> - <li><a href="#library-of-graph-algorithms">Library of Graph Algorithms</a></li> - <li><a href="#use-case-music-profiles">Use-Case: Music Profiles</a></li> - <li><a href="#ongoing-and-future-work">Ongoing and Future Work</a></li> -</ol> - -<p><a href="#top"></a></p> - -<h2 id="what-is-gelly">What is Gelly?</h2> - -<p>Gelly is a Graph API for Flink. It is currently supported in both Java and Scala. -The Scala methods are implemented as wrappers on top of the basic Java operations. -The API contains a set of utility functions for graph analysis, supports iterative graph -processing and introduces a library of graph algorithms.</p> - -<center> -<img src="/img/blog/flink-stack.png" style="width:90%;margin:15px" /> -</center> - -<p><a href="#top">Back to top</a></p> - -<h2 id="graph-representation-and-creation">Graph Representation and Creation</h2> - -<p>In Gelly, a graph is represented by a DataSet of vertices and a DataSet of edges. -A vertex is defined by its unique ID and a value, whereas an edge is defined by its source ID, -target ID, and value. A vertex or edge for which a value is not specified will simply have the -value type set to <code>NullValue</code>.</p> - -<p>A graph can be created from:</p> - -<ol> - <li><strong>DataSet of edges</strong> and an optional <strong>DataSet of vertices</strong> using <code>Graph.fromDataSet()</code></li> - <li><strong>DataSet of Tuple3</strong> and an optional <strong>DataSet of Tuple2</strong> using <code>Graph.fromTupleDataSet()</code></li> - <li><strong>Collection of edges</strong> and an optional <strong>Collection of vertices</strong> using <code>Graph.fromCollection()</code></li> -</ol> - -<p>In all three cases, if the vertices are not provided, -Gelly will automatically produce the vertex IDs from the edge source and target IDs.</p> - -<p><a href="#top">Back to top</a></p> - -<h2 id="transformations-and-utilities">Transformations and Utilities</h2> - -<p>These are methods of the Graph class and include common graph metrics, transformations -and mutations as well as neighborhood aggregations.</p> - -<h4 id="common-graph-metrics">Common Graph Metrics</h4> -<p>These methods can be used to retrieve several graph metrics and properties, such as the number -of vertices, edges and the node degrees.</p> - -<h4 id="transformations">Transformations</h4> -<p>The transformation methods enable several Graph operations, using high-level functions similar to -the ones provided by the batch processing API. These transformations can be applied one after the -other, yielding a new Graph after each step, in a fashion similar to operators on DataSets:</p> - -<div class="highlight"><pre><code class="language-java"><span class="n">inputGraph</span><span class="o">.</span><span class="na">getUndirected</span><span class="o">().</span><span class="na">mapEdges</span><span class="o">(</span><span class="k">new</span> <span class="nf">CustomEd [...] - -<p>Transformations can be applied on:</p> - -<ol> - <li><strong>Vertices</strong>: <code>mapVertices</code>, <code>joinWithVertices</code>, <code>filterOnVertices</code>, <code>addVertex</code>, …</li> - <li><strong>Edges</strong>: <code>mapEdges</code>, <code>filterOnEdges</code>, <code>removeEdge</code>, …</li> - <li><strong>Triplets</strong> (source vertex, target vertex, edge): <code>getTriplets</code></li> -</ol> - -<h4 id="neighborhood-aggregations">Neighborhood Aggregations</h4> - -<p>Neighborhood methods allow vertices to perform an aggregation on their first-hop neighborhood. -This provides a vertex-centric view, where each vertex can access its neighboring edges and neighbor values.</p> - -<p><code>reduceOnEdges()</code> provides access to the neighboring edges of a vertex, -i.e. the edge value and the vertex ID of the edge endpoint. In order to also access the -neighboring vertices’ values, one should call the <code>reduceOnNeighbors()</code> function. -The scope of the neighborhood is defined by the EdgeDirection parameter, which can be IN, OUT or ALL, -to gather in-coming, out-going or all edges (neighbors) of a vertex.</p> - -<p>The two neighborhood -functions mentioned above can only be used when the aggregation function is associative and commutative. -In case the function does not comply with these restrictions or if it is desirable to return zero, -one or more values per vertex, the more general <code>groupReduceOnEdges()</code> and -<code>groupReduceOnNeighbors()</code> functions must be called.</p> - -<p>Consider the following graph, for instance:</p> - -<center> -<img src="/img/blog/neighborhood.png" style="width:60%;margin:15px" /> -</center> - -<p>Assume you would want to compute the sum of the values of all incoming neighbors for each vertex. -We will call the <code>reduceOnNeighbors()</code> aggregation method since the sum is an associative and commutative operation and the neighbors’ values are needed:</p> - -<div class="highlight"><pre><code class="language-java"><span class="n">graph</span><span class="o">.</span><span class="na">reduceOnNeighbors</span><span class="o">(</span><span class="k">new</span> <span class="nf">SumValues</span><span class="o">(),</span> <span class="n">EdgeDire [...] - -<p>The vertex with id 1 is the only node that has no incoming edges. The result is therefore:</p> - -<center> -<img src="/img/blog/reduce-on-neighbors.png" style="width:90%;margin:15px" /> -</center> - -<p><a href="#top">Back to top</a></p> - -<h2 id="iterative-graph-processing">Iterative Graph Processing</h2> - -<p>During the past few years, many different programming models for distributed graph processing -have been introduced: <a href="http://delivery.acm.org/10.1145/2490000/2484843/a22-salihoglu.pdf?ip=141.23.53.206&amp;id=2484843&amp;acc=ACTIVE%20SERVICE&amp;key=2BA2C432AB83DA15.0F42380CB8DD3307.4D4702B0C3E38B35.4D4702B0C3E38B35&amp;CFID=706313474&amp;CFTOKEN=60107876&amp;__acm__=1440408958_b131e035942130653e5782409b5c0cde">vertex-centric</a>, -<a href="http://researcher.ibm.com/researcher/files/us-ytian/giraph++.pdf">partition-centric</a>, <a href="http://www.eecs.harvard.edu/cs261/notes/gonzalez-2012.htm">gather-apply-scatter</a>, -<a href="http://infoscience.epfl.ch/record/188535/files/paper.pdf">edge-centric</a>, <a href="http://www.vldb.org/pvldb/vol7/p1673-quamar.pdf">neighborhood-centric</a>. -Each one of these models targets a specific class of graph applications and each corresponding -system implementation optimizes the runtime respectively. In Gelly, we would like to exploit the -flexible dataflow model and the efficient iterations of Flink, to support multiple distributed -graph processing models on top of the same system.</p> - -<p>Currently, Gelly has methods for writing vertex-centric programs and provides support for programs -implemented using the gather-sum(accumulate)-apply model. We are also considering to offer support -for the partition-centric computation model, using Fink’s <code>mapPartition()</code> operator. -This model exposes the partition structure to the user and allows local graph structure exploitation -inside a partition to avoid unnecessary communication.</p> - -<h4 id="vertex-centric">Vertex-centric</h4> - -<p>Gelly wraps Flink’s <a href="https://ci.apache.org/projects/flink/flink-docs-release-0.8/spargel_guide.html">Spargel APi</a> to -support the vertex-centric, Pregel-like programming model. Gelly’s <code>runVertexCentricIteration</code> method accepts two user-defined functions:</p> - -<ol> - <li><strong>MessagingFunction:</strong> defines what messages a vertex sends out for the next superstep.</li> - <li><strong>VertexUpdateFunction:</strong>* defines how a vertex will update its value based on the received messages.</li> -</ol> - -<p>The method will execute the vertex-centric iteration on the input Graph and return a new Graph, with updated vertex values.</p> - -<p>Gelly’s vertex-centric programming model exploits Flink’s efficient delta iteration operators. -Many iterative graph algorithms expose non-uniform behavior, where some vertices converge to -their final value faster than others. In such cases, the number of vertices that need to be -recomputed during an iteration decreases as the algorithm moves towards convergence.</p> - -<p>For example, consider a Single Source Shortest Paths problem on the following graph, where S -is the source node, i is the iteration counter and the edge values represent distances between nodes:</p> - -<center> -<img src="/img/blog/sssp.png" style="width:90%;margin:15px" /> -</center> - -<p>In each iteration, a vertex receives distances from its neighbors and adopts the minimum of -these distances and its current distance as the new value. Then, it propagates its new value -to its neighbors. If a vertex does not change value during an iteration, there is no need for -it to propagate its old distance to its neighbors; as they have already taken it into account.</p> - -<p>Flink’s <code>IterateDelta</code> operator permits exploitation of this property as well as the -execution of computations solely on the active parts of the graph. The operator receives two inputs:</p> - -<ol> - <li>the <strong>Solution Set</strong>, which represents the current state of the input and</li> - <li>the <strong>Workset</strong>, which determines which parts of the graph will be recomputed in the next iteration.</li> -</ol> - -<p>In the SSSP example above, the Workset contains the vertices which update their distances. -The user-defined iterative function is applied on these inputs to produce state updates. -These updates are efficiently applied on the state, which is kept in memory.</p> - -<center> -<img src="/img/blog/iteration.png" style="width:60%;margin:15px" /> -</center> - -<p>Internally, a vertex-centric iteration is a Flink delta iteration, where the initial Solution Set -is the vertex set of the input graph and the Workset is created by selecting the active vertices, -i.e. the ones that updated their value in the previous iteration. The messaging and vertex-update -functions are user-defined functions wrapped inside coGroup operators. In each superstep, -the active vertices (Workset) are coGrouped with the edges to generate the neighborhoods for -each vertex. The messaging function is then applied on each neighborhood. Next, the result of the -messaging function is coGrouped with the current vertex values (Solution Set) and the user-defined -vertex-update function is applied on the result. The output of this coGroup operator is finally -used to update the Solution Set and create the Workset input for the next iteration.</p> - -<center> -<img src="/img/blog/vertex-centric-plan.png" style="width:40%;margin:15px" /> -</center> - -<h4 id="gather-sum-apply">Gather-Sum-Apply</h4> - -<p>Gelly supports a variation of the popular Gather-Sum-Apply-Scatter computation model, -introduced by PowerGraph. In GSA, a vertex pulls information from its neighbors as opposed to the -vertex-centric approach where the updates are pushed from the incoming neighbors. -The <code>runGatherSumApplyIteration()</code> accepts three user-defined functions:</p> - -<ol> - <li><strong>GatherFunction:</strong> gathers neighboring partial values along in-edges.</li> - <li><strong>SumFunction:</strong> accumulates/reduces the values into a single one.</li> - <li><strong>ApplyFunction:</strong> uses the result computed in the sum phase to update the current vertex’s value.</li> -</ol> - -<p>Similarly to vertex-centric, GSA leverages Flink’s delta iteration operators as, in many cases, -vertex values do not need to be recomputed during an iteration.</p> - -<p>Let us reconsider the Single Source Shortest Paths algorithm. In each iteration, a vertex:</p> - -<ol> - <li><strong>Gather</strong> retrieves distances from its neighbors summed up with the corresponding edge values;</li> - <li><strong>Sum</strong> compares the newly obtained distances in order to extract the minimum;</li> - <li><strong>Apply</strong> and finally adopts the minimum distance computed in the sum step, -provided that it is lower than its current value. If a vertex’s value does not change during -an iteration, it no longer propagates its distance.</li> -</ol> - -<p>Internally, a Gather-Sum-Apply Iteration is a Flink delta iteration where the initial solution -set is the vertex input set and the workset is created by selecting the active vertices.</p> - -<p>The three functions: gather, sum and apply are user-defined functions wrapped in map, reduce -and join operators respectively. In each superstep, the active vertices are joined with the -edges in order to create neighborhoods for each vertex. The gather function is then applied on -the neighborhood values via a map function. Afterwards, the result is grouped by the vertex ID -and reduced using the sum function. Finally, the outcome of the sum phase is joined with the -current vertex values (solution set), the values are updated, thus creating a new workset that -serves as input for the next iteration.</p> - -<center> -<img src="/img/blog/GSA-plan.png" style="width:40%;margin:15px" /> -</center> - -<p><a href="#top">Back to top</a></p> - -<h2 id="library-of-graph-algorithms">Library of Graph Algorithms</h2> - -<p>We are building a library of graph algorithms in Gelly, to easily analyze large-scale graphs. -These algorithms extend the <code>GraphAlgorithm</code> interface and can be simply executed on -the input graph by calling a <code>run()</code> method.</p> - -<p>We currently have implementations of the following algorithms:</p> - -<ol> - <li>PageRank</li> - <li>Single-Source-Shortest-Paths</li> - <li>Label Propagation</li> - <li>Community Detection (based on <a href="http://arxiv.org/pdf/0808.2633.pdf">this paper</a>)</li> - <li>Connected Components</li> - <li>GSA Connected Components</li> - <li>GSA PageRank</li> - <li>GSA Single-Source-Shortest-Paths</li> -</ol> - -<p>Gelly also offers implementations of common graph algorithms through <a href="https://github.com/apache/flink/tree/master/flink-staging/flink-gelly/src/main/java/org/apache/flink/graph/example">examples</a>. -Among them, one can find graph weighting schemes, like Jaccard Similarity and Euclidean Distance Weighting, -as well as computation of common graph metrics.</p> - -<p><a href="#top">Back to top</a></p> - -<h2 id="use-case-music-profiles">Use-Case: Music Profiles</h2> - -<p>In the following section, we go through a use-case scenario that combines the Flink DataSet API -with Gelly in order to process users’ music preferences to suggest additions to their playlist.</p> - -<p>First, we read a user’s music profile which is in the form of user-id, song-id and the number of -plays that each song has. We then filter out the list of songs the users do not wish to see in their -playlist. Then we compute the top songs per user (i.e. the songs a user listened to the most). -Finally, as a separate use-case on the same data set, we create a user-user similarity graph based -on the common songs and use this resulting graph to detect communities by calling Gelly’s Label Propagation -library method.</p> - -<p>For running the example implementation, please use the 0.10-SNAPSHOT version of Flink as a -dependency. The full example code base can be found <a href="https://github.com/apache/flink/blob/master/flink-staging/flink-gelly/src/main/java/org/apache/flink/graph/example/MusicProfiles.java">here</a>. The public data set used for testing -can be found <a href="http://labrosa.ee.columbia.edu/millionsong/tasteprofile">here</a>. This data set contains <strong>48,373,586</strong> real user-id, song-id and -play-count triplets.</p> - -<p><strong>Note:</strong> The code snippets in this post try to reduce verbosity by skipping type parameters of generic functions. Please have a look at <a href="https://github.com/apache/flink/blob/master/flink-staging/flink-gelly/src/main/java/org/apache/flink/graph/example/MusicProfiles.java">the full example</a> for the correct and complete code.</p> - -<h4 id="filtering-out-bad-records">Filtering out Bad Records</h4> - -<p>After reading the <code>(user-id, song-id, play-count)</code> triplets from a CSV file and after parsing a -text file in order to retrieve the list of songs that a user would not want to include in a -playlist, we use a coGroup function to filter out the mismatches.</p> - -<div class="highlight"><pre><code class="language-java"><span class="c1">// read the user-song-play triplets.</span> -<span class="n">DataSet</span><span class="o">&lt;</span><span class="n">Tuple3</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&am [...] - <span class="n">getUserSongTripletsData</span><span class="o">(</span><span class="n">env</span><span class="o">);</span> - -<span class="c1">// read the mismatches dataset and extract the songIDs</span> -<span class="n">DataSet</span><span class="o">&lt;</span><span class="n">Tuple3</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&am [...] - <span class="o">.</span><span class="na">coGroup</span><span class="o">(</span><span class="n">mismatches</span><span class="o">).</span><span class="na">where</span><span class="o">(</span><span class="mi">1</span><span class="o">).</span><span class="na">equalTo</s [...] - <span class="o">.</span><span class="na">with</span><span class="o">(</span><span class="k">new</span> <span class="nf">CoGroupFunction</span><span class="o">()</span> <span class="o">{</span> - <span class="kt">void</span> <span class="nf">coGroup</span><span class="o">(</span><span class="n">Iterable</span> <span class="n">triplets</span><span class="o">,</span> <span class="n">Iterable</span> <span class="n">invalidSongs</span><span class="o">,</span> <span class [...] - <span class="k">if</span> <span class="o">(!</span><span class="n">invalidSongs</span><span class="o">.</span><span class="na">iterator</span><span class="o">().</span><span class="na">hasNext</span><span class="o">())</span> <span class="o">{</span> - <span class="k">for</span> <span class="o">(</span><span class="n">Tuple3</span> <span class="n">triplet</span> <span class="o">:</span> <span class="n">triplets</span><span class="o">)</span> <span class="o">{</span> <span class="c1">// valid triplet</span> - <span class="n">out</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="n">triplet</span><span class="o">);</span> - <span class="o">}</span> - <span class="o">}</span> - <span class="o">}</span> - <span class="o">}</span></code></pre></div> - -<p>The coGroup simply takes the triplets whose song-id (second field) matches the song-id from the -mismatches list (first field) and if the iterator was empty for a certain triplet, meaning that -there were no mismatches found, the triplet associated with that song is collected.</p> - -<h4 id="compute-the-top-songs-per-user">Compute the Top Songs per User</h4> - -<p>As a next step, we would like to see which songs a user played more often. To this end, we -build a user-song weighted, bipartite graph in which edge source vertices are users, edge target -vertices are songs and where the weight represents the number of times the user listened to that -certain song.</p> - -<center> -<img src="/img/blog/user-song-graph.png" style="width:90%;margin:15px" /> -</center> - -<div class="highlight"><pre><code class="language-java"><span class="c1">// create a user -&gt; song weighted bipartite graph where the edge weights</span> -<span class="c1">// correspond to play counts</span> -<span class="n">Graph</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">NullValue</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;</span> <span class="n">userSongGraph</span> <span class="o&qu [...] - -<p>Consult the <a href="https://ci.apache.org/projects/flink/flink-docs-master/dev/libs/gelly/">Gelly guide</a> for guidelines -on how to create a graph from a given DataSet of edges or from a collection.</p> - -<p>To retrieve the top songs per user, we call the groupReduceOnEdges function as it perform an -aggregation over the first hop neighborhood taking just the edges into consideration. We will -basically iterate through the edge value and collect the target (song) of the maximum weight edge.</p> - -<div class="highlight"><pre><code class="language-java"><span class="c1">//get the top track (most listened to) for each user</span> -<span class="n">DataSet</span><span class="o">&lt;</span><span class="n">Tuple2</span><span class="o">&gt;</span> <span class="n">usersWithTopTrack</span> <span class="o">=</span> <span class="n">userSongGraph</span> - <span class="o">.</span><span class="na">groupReduceOnEdges</span><span class="o">(</span><span class="k">new</span> <span class="nf">GetTopSongPerUser</span><span class="o">(),</span> <span class="n">EdgeDirection</span><span class="o">.</span><span class="na">OUT</span><span class [...] - -<span class="kd">class</span> <span class="nc">GetTopSongPerUser</span> <span class="kd">implements</span> <span class="n">EdgesFunctionWithVertexValue</span> <span class="o">{</span> - <span class="kt">void</span> <span class="nf">iterateEdges</span><span class="o">(</span><span class="n">Vertex</span> <span class="n">vertex</span><span class="o">,</span> <span class="n">Iterable</span><span class="o">&lt;</span><span class="n">Edge</span><span class="o" [...] - <span class="kt">int</span> <span class="n">maxPlaycount</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> - <span class="n">String</span> <span class="n">topSong</span> <span class="o">=</span> <span class="s">&quot;&quot;</span><span class="o">;</span> - - <span class="k">for</span> <span class="o">(</span><span class="n">Edge</span> <span class="n">edge</span> <span class="o">:</span> <span class="n">edges</span><span class="o">)</span> <span class="o">{</span> - <span class="k">if</span> <span class="o">(</span><span class="n">edge</span><span class="o">.</span><span class="na">getValue</span><span class="o">()</span> <span class="o">&gt;</span> <span class="n">maxPlaycount</span><span class="o">)</span> <span class="o"> [...] - <span class="n">maxPlaycount</span> <span class="o">=</span> <span class="n">edge</span><span class="o">.</span><span class="na">getValue</span><span class="o">();</span> - <span class="n">topSong</span> <span class="o">=</span> <span class="n">edge</span><span class="o">.</span><span class="na">getTarget</span><span class="o">();</span> - <span class="o">}</span> - <span class="o">}</span> - <span class="k">return</span> <span class="k">new</span> <span class="nf">Tuple2</span><span class="o">(</span><span class="n">vertex</span><span class="o">.</span><span class="na">getId</span><span class="o">(),</span> <span class="n">topSong</span><span class="o">);&l [...] - <span class="o">}</span> -<span class="o">}</span></code></pre></div> - -<h4 id="creating-a-user-user-similarity-graph">Creating a User-User Similarity Graph</h4> - -<p>Clustering users based on common interests, in this case, common top songs, could prove to be -very useful for advertisements or for recommending new musical compilations. In a user-user graph, -two users who listen to the same song will simply be linked together through an edge as depicted -in the figure below.</p> - -<center> -<img src="/img/blog/user-song-to-user-user.png" style="width:90%;margin:15px" /> -</center> - -<p>To form the user-user graph in Flink, we will simply take the edges from the user-song graph -(left-hand side of the image), group them by song-id, and then add all the users (source vertex ids) -to an ArrayList.</p> - -<p>We then match users who listened to the same song two by two, creating a new edge to mark their -common interest (right-hand side of the image).</p> - -<p>Afterwards, we perform a <code>distinct()</code> operation to avoid creation of duplicate data. -Considering that we now have the DataSet of edges which present interest, creating a graph is as -straightforward as a call to the <code>Graph.fromDataSet()</code> method.</p> - -<div class="highlight"><pre><code class="language-java"><span class="c1">// create a user-user similarity graph:</span> -<span class="c1">// two users that listen to the same song are connected</span> -<span class="n">DataSet</span><span class="o">&lt;</span><span class="n">Edge</span><span class="o">&gt;</span> <span class="n">similarUsers</span> <span class="o">=</span> <span class="n">userSongGraph</span><span class="o">.</span><span class="na">getEdges</span><span class=" [...] - <span class="c1">// filter out user-song edges that are below the playcount threshold</span> - <span class="o">.</span><span class="na">filter</span><span class="o">(</span><span class="k">new</span> <span class="n">FilterFunction</span><span class="o">&lt;</span><span class="n">Edge</span><span class="o">&lt;</span><span class="n">String</span><span class="o&quo [...] - <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">filter</span><span class="o">(</span><span class="n">Edge</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o [...] - <span class="k">return</span> <span class="o">(</span><span class="n">edge</span><span class="o">.</span><span class="na">getValue</span><span class="o">()</span> <span class="o">&gt;</span> <span class="n">playcountThreshold</span><span class="o">);</span> - <span class="o">}</span> - <span class="o">})</span> - <span class="o">.</span><span class="na">groupBy</span><span class="o">(</span><span class="mi">1</span><span class="o">)</span> - <span class="o">.</span><span class="na">reduceGroup</span><span class="o">(</span><span class="k">new</span> <span class="nf">GroupReduceFunction</span><span class="o">()</span> <span class="o">{</span> - <span class="kt">void</span> <span class="nf">reduce</span><span class="o">(</span><span class="n">Iterable</span><span class="o">&lt;</span><span class="n">Edge</span><span class="o">&gt;</span> <span class="n">edges</span><span class="o">,</span> <span class=" [...] - <span class="n">List</span> <span class="n">users</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">ArrayList</span><span class="o">();</span> - <span class="k">for</span> <span class="o">(</span><span class="n">Edge</span> <span class="n">edge</span> <span class="o">:</span> <span class="n">edges</span><span class="o">)</span> <span class="o">{</span> - <span class="n">users</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">edge</span><span class="o">.</span><span class="na">getSource</span><span class="o">());</span> - <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">us [...] - <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">j</span> <span class="o">=</span> <span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="o">;</span> <span class="n">j</s [...] - <span class="n">out</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="k">new</span> <span class="nf">Edge</span><span class="o">(</span><span class="n">users</span><span class="o">.</span><span class="na&qu [...] - <span class="o">}</span> - <span class="o">}</span> - <span class="o">}</span> - <span class="o">}</span> - <span class="o">})</span> - <span class="o">.</span><span class="na">distinct</span><span class="o">();</span> - -<span class="n">Graph</span> <span class="n">similarUsersGraph</span> <span class="o">=</span> <span class="n">Graph</span><span class="o">.</span><span class="na">fromDataSet</span><span class="o">(</span><span class="n">similarUsers</span><span class="o">).</span><span class="na" [...] - -<p>After having created a user-user graph, it would make sense to detect the various communities -formed. To do so, we first initialize each vertex with a numeric label using the -<code>joinWithVertices()</code> function that takes a data set of Tuple2 as a parameter and joins -the id of a vertex with the first element of the tuple, afterwards applying a map function. -Finally, we call the <code>run()</code> method with the LabelPropagation library method passed -as a parameter. In the end, the vertices will be updated to contain the most frequent label -among their neighbors.</p> - -<div class="highlight"><pre><code class="language-java"><span class="c1">// detect user communities using label propagation</span> -<span class="c1">// initialize each vertex with a unique numeric label</span> -<span class="n">DataSet</span><span class="o">&lt;</span><span class="n">Tuple2</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;&gt;</span> <span class="n">idsWithInitialLabels</span> <span [...] - <span class="o">.</span><span class="na">zipWithUniqueId</span><span class="o">(</span><span class="n">similarUsersGraph</span><span class="o">.</span><span class="na">getVertexIds</span><span class="o">())</span> - <span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="k">new</span> <span class="n">MapFunction</span><span class="o">&lt;</span><span class="n">Tuple2</span><span class="o">&lt;</span><span class="n">Long</span><span class="o"> [...] - <span class="nd">@Override</span> - <span class="kd">public</span> <span class="n">Tuple2</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;</span> <span class="nf">map</span><span class="o">(</span><span class="n& [...] - <span class="k">return</span> <span class="k">new</span> <span class="n">Tuple2</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;(</span><span class="n">tuple2</span><span class= [...] - <span class="o">}</span> - <span class="o">});</span> - -<span class="c1">// update the vertex values and run the label propagation algorithm</span> -<span class="n">DataSet</span><span class="o">&lt;</span><span class="n">Vertex</span><span class="o">&gt;</span> <span class="n">verticesWithCommunity</span> <span class="o">=</span> <span class="n">similarUsersGraph</span> - <span class="o">.</span><span class="na">joinWithVertices</span><span class="o">(</span><span class="n">idsWithlLabels</span><span class="o">,</span> <span class="k">new</span> <span class="nf">MapFunction</span><span class="o">()</span> <span class="o">{</span> - <span class="kd">public</span> <span class="n">Long</span> <span class="nf">map</span><span class="o">(</span><span class="n">Tuple2</span> <span class="n">idWithLabel</span><span class="o">)</span> <span class="o">{</span> - <span class="k">return</span> <span class="n">idWithLabel</span><span class="o">.</span><span class="na">f1</span><span class="o">;</span> - <span class="o">}</span> - <span class="o">})</span> - <span class="o">.</span><span class="na">run</span><span class="o">(</span><span class="k">new</span> <span class="nf">LabelPropagation</span><span class="o">(</span><span class="n">numIterations</span><span class="o">))</span> - <span class="o">.</span><span class="na">getVertices</span><span class="o">();</span></code></pre></div> - -<p><a href="#top">Back to top</a></p> - -<h2 id="ongoing-and-future-work">Ongoing and Future Work</h2> - -<p>Currently, Gelly matches the basic functionalities provided by most state-of-the-art graph -processing systems. Our vision is to turn Gelly into more than “yet another library for running -PageRank-like algorithms” by supporting generic iterations, implementing graph partitioning, -providing bipartite graph support and by offering numerous other features.</p> - -<p>We are also enriching Flink Gelly with a set of operators suitable for highly skewed graphs -as well as a Graph API built on Flink Streaming.</p> - -<p>In the near future, we would like to see how Gelly can be integrated with graph visualization -tools, graph database systems and sampling techniques.</p> - -<p>Curious? Read more about our plans for Gelly in the <a href="https://cwiki.apache.org/confluence/display/FLINK/Flink+Gelly">roadmap</a>.</p> - -<p><a href="#top">Back to top</a></p> - -<h2 id="links">Links</h2> -<p><a href="https://ci.apache.org/projects/flink/flink-docs-master/dev/libs/gelly/">Gelly Documentation</a></p> -</description> -<pubDate>Mon, 24 Aug 2015 00:00:00 +0200</pubDate> -<link>https://flink.apache.org/news/2015/08/24/introducing-flink-gelly.html</link> -<guid isPermaLink="true">/news/2015/08/24/introducing-flink-gelly.html</guid> -</item> - </channel> </rss> diff --git a/content/blog/index.html b/content/blog/index.html index fa55ee8..c75001f 100644 --- a/content/blog/index.html +++ b/content/blog/index.html @@ -196,6 +196,19 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2020/07/06/release-1.11.0.html">Apache Flink 1.11.0 Release Announcement</a></h2> + + <p>06 Jul 2020 + Marta Paes (<a href="https://twitter.com/morsapaes";>@morsapaes</a>)</p> + + <p>The Apache Flink community is proud to announce the release of Flink 1.11.0! More than 200 contributors worked on over 1.3k issues to bring significant improvements to usability as well as new features to Flink users across the whole API stack. We're particularly excited about unaligned checkpoints to cope with high backpressure scenarios, a new source API that simplifies and unifies the implementation of (custom) sources, and support for Change Data Capture (CDC) and other comm [...] + + <p><a href="/news/2020/07/06/release-1.11.0.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 2</a></h2> <p>23 Jun 2020 @@ -325,19 +338,6 @@ and provide a tutorial for running Streaming ETL with Flink on Zeppelin.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></h2> - - <p>15 Apr 2020 - Nico Kruber </p> - - <p>Serialization is a crucial element of your Flink job. This article is the first in a series of posts that will highlight Flink’s serialization stack, and looks at the different ways Flink can serialize your data types.</p> - - <p><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -370,6 +370,16 @@ and provide a tutorial for running Streaming ETL with Flink on Zeppelin.</p> <ul id="markdown-toc"> + <li><a href="/news/2020/07/06/release-1.11.0.html">Apache Flink 1.11.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 2</a></li> diff --git a/content/blog/page10/index.html b/content/blog/page10/index.html index 5bc32c9..9da2914 100644 --- a/content/blog/page10/index.html +++ b/content/blog/page10/index.html @@ -196,6 +196,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2016/03/08/release-1.0.0.html">Announcing Apache Flink 1.0.0</a></h2> + + <p>08 Mar 2016 + </p> + + <p><p>The Apache Flink community is pleased to announce the availability of the 1.0.0 release. The community put significant effort into improving and extending Apache Flink since the last release, focusing on improving the experience of writing and executing data stream processing pipelines in production.</p> + +</p> + + <p><a href="/news/2016/03/08/release-1.0.0.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2016/02/11/release-0.10.2.html">Flink 0.10.2 Released</a></h2> <p>11 Feb 2016 @@ -328,24 +343,6 @@ Apache Flink started.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2015/08/24/introducing-flink-gelly.html">Introducing Gelly: Graph Processing with Apache Flink</a></h2> - - <p>24 Aug 2015 - </p> - - <p><p>This blog post introduces <strong>Gelly</strong>, Apache Flink’s <em>graph-processing API and library</em>. Flink’s native support -for iterations makes it a suitable platform for large-scale graph analytics. -By leveraging delta iterations, Gelly is able to map various graph processing models such as -vertex-centric or gather-sum-apply to Flink dataflows.</p> - -</p> - - <p><a href="/news/2015/08/24/introducing-flink-gelly.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -378,6 +375,16 @@ vertex-centric or gather-sum-apply to Flink dataflows.</p> <ul id="markdown-toc"> + <li><a href="/news/2020/07/06/release-1.11.0.html">Apache Flink 1.11.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 2</a></li> diff --git a/content/blog/page11/index.html b/content/blog/page11/index.html index 2b4dfdf..ce6d1cc 100644 --- a/content/blog/page11/index.html +++ b/content/blog/page11/index.html @@ -196,6 +196,24 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2015/08/24/introducing-flink-gelly.html">Introducing Gelly: Graph Processing with Apache Flink</a></h2> + + <p>24 Aug 2015 + </p> + + <p><p>This blog post introduces <strong>Gelly</strong>, Apache Flink’s <em>graph-processing API and library</em>. Flink’s native support +for iterations makes it a suitable platform for large-scale graph analytics. +By leveraging delta iterations, Gelly is able to map various graph processing models such as +vertex-centric or gather-sum-apply to Flink dataflows.</p> + +</p> + + <p><a href="/news/2015/08/24/introducing-flink-gelly.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2015/06/24/announcing-apache-flink-0.9.0-release.html">Announcing Apache Flink 0.9.0</a></h2> <p>24 Jun 2015 @@ -337,21 +355,6 @@ and offers a new API including definition of flexible windows.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2015/01/21/release-0.8.html">Apache Flink 0.8.0 available</a></h2> - - <p>21 Jan 2015 - </p> - - <p><p>We are pleased to announce the availability of Flink 0.8.0. This release includes new user-facing features as well as performance and bug fixes, extends the support for filesystems and introduces the Scala API and flexible windowing semantics for Flink Streaming. A total of 33 people have contributed to this release, a big thanks to all of them!</p> - -</p> - - <p><a href="/news/2015/01/21/release-0.8.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -384,6 +387,16 @@ and offers a new API including definition of flexible windows.</p> <ul id="markdown-toc"> + <li><a href="/news/2020/07/06/release-1.11.0.html">Apache Flink 1.11.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 2</a></li> diff --git a/content/blog/page12/index.html b/content/blog/page12/index.html index 8d3a1d2..27f2425 100644 --- a/content/blog/page12/index.html +++ b/content/blog/page12/index.html @@ -196,6 +196,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2015/01/21/release-0.8.html">Apache Flink 0.8.0 available</a></h2> + + <p>21 Jan 2015 + </p> + + <p><p>We are pleased to announce the availability of Flink 0.8.0. This release includes new user-facing features as well as performance and bug fixes, extends the support for filesystems and introduces the Scala API and flexible windowing semantics for Flink Streaming. A total of 33 people have contributed to this release, a big thanks to all of them!</p> + +</p> + + <p><a href="/news/2015/01/21/release-0.8.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2015/01/06/december-in-flink.html">December 2014 in the Flink community</a></h2> <p>06 Jan 2015 @@ -320,6 +335,16 @@ academic and open source project that Flink originates from.</p> <ul id="markdown-toc"> + <li><a href="/news/2020/07/06/release-1.11.0.html">Apache Flink 1.11.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 2</a></li> diff --git a/content/blog/page2/index.html b/content/blog/page2/index.html index bad923d..f634033 100644 --- a/content/blog/page2/index.html +++ b/content/blog/page2/index.html @@ -196,6 +196,19 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></h2> + + <p>15 Apr 2020 + Nico Kruber </p> + + <p>Serialization is a crucial element of your Flink job. This article is the first in a series of posts that will highlight Flink’s serialization stack, and looks at the different ways Flink can serialize your data types.</p> + + <p><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python Support for UDFs in Flink's Table API</a></h2> <p>09 Apr 2020 @@ -319,21 +332,6 @@ This release marks a big milestone: Stateful Functions 2.0 is not only an API up <hr> - <article> - <h2 class="blog-title"><a href="/news/2020/01/30/release-1.9.2.html">Apache Flink 1.9.2 Released</a></h2> - - <p>30 Jan 2020 - Hequn Cheng (<a href="https://twitter.com/HequnC";>@HequnC</a>)</p> - - <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.9 series.</p> - -</p> - - <p><a href="/news/2020/01/30/release-1.9.2.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -366,6 +364,16 @@ This release marks a big milestone: Stateful Functions 2.0 is not only an API up <ul id="markdown-toc"> + <li><a href="/news/2020/07/06/release-1.11.0.html">Apache Flink 1.11.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 2</a></li> diff --git a/content/blog/page3/index.html b/content/blog/page3/index.html index 7de6587..1a1760b 100644 --- a/content/blog/page3/index.html +++ b/content/blog/page3/index.html @@ -196,6 +196,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2020/01/30/release-1.9.2.html">Apache Flink 1.9.2 Released</a></h2> + + <p>30 Jan 2020 + Hequn Cheng (<a href="https://twitter.com/HequnC";>@HequnC</a>)</p> + + <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.9 series.</p> + +</p> + + <p><a href="/news/2020/01/30/release-1.9.2.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State Unlocked: Interacting with State in Apache Flink</a></h2> <p>29 Jan 2020 @@ -318,22 +333,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2019/08/22/release-1.9.0.html">Apache Flink 1.9.0 Release Announcement</a></h2> - - <p>22 Aug 2019 - </p> - - <p><p>The Apache Flink community is proud to announce the release of Apache Flink -1.9.0.</p> - -</p> - - <p><a href="/news/2019/08/22/release-1.9.0.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -366,6 +365,16 @@ <ul id="markdown-toc"> + <li><a href="/news/2020/07/06/release-1.11.0.html">Apache Flink 1.11.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 2</a></li> diff --git a/content/blog/page4/index.html b/content/blog/page4/index.html index 6d53835..f3b4b2f 100644 --- a/content/blog/page4/index.html +++ b/content/blog/page4/index.html @@ -196,6 +196,22 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2019/08/22/release-1.9.0.html">Apache Flink 1.9.0 Release Announcement</a></h2> + + <p>22 Aug 2019 + </p> + + <p><p>The Apache Flink community is proud to announce the release of Apache Flink +1.9.0.</p> + +</p> + + <p><a href="/news/2019/08/22/release-1.9.0.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/2019/07/23/flink-network-stack-2.html">Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing</a></h2> <p>23 Jul 2019 @@ -322,19 +338,6 @@ for more details.</p> <hr> - <article> - <h2 class="blog-title"><a href="/features/2019/03/11/prometheus-monitoring.html">Flink and Prometheus: Cloud-native monitoring of streaming applications</a></h2> - - <p>11 Mar 2019 - Maximilian Bode, TNG Technology Consulting (<a href="https://twitter.com/mxpbode";>@mxpbode</a>)</p> - - <p>This blog post describes how developers can leverage Apache Flink's built-in metrics system together with Prometheus to observe and monitor streaming applications in an effective way.</p> - - <p><a href="/features/2019/03/11/prometheus-monitoring.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -367,6 +370,16 @@ for more details.</p> <ul id="markdown-toc"> + <li><a href="/news/2020/07/06/release-1.11.0.html">Apache Flink 1.11.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 2</a></li> diff --git a/content/blog/page5/index.html b/content/blog/page5/index.html index c970070..85fc481 100644 --- a/content/blog/page5/index.html +++ b/content/blog/page5/index.html @@ -196,6 +196,19 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/features/2019/03/11/prometheus-monitoring.html">Flink and Prometheus: Cloud-native monitoring of streaming applications</a></h2> + + <p>11 Mar 2019 + Maximilian Bode, TNG Technology Consulting (<a href="https://twitter.com/mxpbode";>@mxpbode</a>)</p> + + <p>This blog post describes how developers can leverage Apache Flink's built-in metrics system together with Prometheus to observe and monitor streaming applications in an effective way.</p> + + <p><a href="/features/2019/03/11/prometheus-monitoring.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2019/03/06/ffsf-preview.html">What to expect from Flink Forward San Francisco 2019</a></h2> <p>06 Mar 2019 @@ -326,21 +339,6 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa <hr> - <article> - <h2 class="blog-title"><a href="/news/2018/10/29/release-1.6.2.html">Apache Flink 1.6.2 Released</a></h2> - - <p>29 Oct 2018 - </p> - - <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.6 series.</p> - -</p> - - <p><a href="/news/2018/10/29/release-1.6.2.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -373,6 +371,16 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa <ul id="markdown-toc"> + <li><a href="/news/2020/07/06/release-1.11.0.html">Apache Flink 1.11.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 2</a></li> diff --git a/content/blog/page6/index.html b/content/blog/page6/index.html index 9b8bdd5..8d3489c 100644 --- a/content/blog/page6/index.html +++ b/content/blog/page6/index.html @@ -196,6 +196,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2018/10/29/release-1.6.2.html">Apache Flink 1.6.2 Released</a></h2> + + <p>29 Oct 2018 + </p> + + <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.6 series.</p> + +</p> + + <p><a href="/news/2018/10/29/release-1.6.2.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2018/10/29/release-1.5.5.html">Apache Flink 1.5.5 Released</a></h2> <p>29 Oct 2018 @@ -330,21 +345,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2018/03/08/release-1.4.2.html">Apache Flink 1.4.2 Released</a></h2> - - <p>08 Mar 2018 - </p> - - <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.4 series.</p> - -</p> - - <p><a href="/news/2018/03/08/release-1.4.2.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -377,6 +377,16 @@ <ul id="markdown-toc"> + <li><a href="/news/2020/07/06/release-1.11.0.html">Apache Flink 1.11.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 2</a></li> diff --git a/content/blog/page7/index.html b/content/blog/page7/index.html index 7aa5f1d..0c80454 100644 --- a/content/blog/page7/index.html +++ b/content/blog/page7/index.html @@ -196,6 +196,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2018/03/08/release-1.4.2.html">Apache Flink 1.4.2 Released</a></h2> + + <p>08 Mar 2018 + </p> + + <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.4 series.</p> + +</p> + + <p><a href="/news/2018/03/08/release-1.4.2.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/features/2018/03/01/end-to-end-exactly-once-apache-flink.html">An Overview of End-to-End Exactly-Once Processing in Apache Flink (with Apache Kafka, too!)</a></h2> <p>01 Mar 2018 @@ -327,21 +342,6 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community <hr> - <article> - <h2 class="blog-title"><a href="/news/2017/06/01/release-1.3.0.html">Apache Flink 1.3.0 Release Announcement</a></h2> - - <p>01 Jun 2017 by Robert Metzger (<a href="https://twitter.com/";>@rmetzger_</a>) - </p> - - <p><p>The Apache Flink community is pleased to announce the 1.3.0 release. Over the past 4 months, the Flink community has been working hard to resolve more than 680 issues. See the <a href="/blog/release_1.3.0-changelog.html">complete changelog</a> for more detail.</p> - -</p> - - <p><a href="/news/2017/06/01/release-1.3.0.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -374,6 +374,16 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community <ul id="markdown-toc"> + <li><a href="/news/2020/07/06/release-1.11.0.html">Apache Flink 1.11.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 2</a></li> diff --git a/content/blog/page8/index.html b/content/blog/page8/index.html index 0b5facc..c5e623e 100644 --- a/content/blog/page8/index.html +++ b/content/blog/page8/index.html @@ -196,6 +196,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2017/06/01/release-1.3.0.html">Apache Flink 1.3.0 Release Announcement</a></h2> + + <p>01 Jun 2017 by Robert Metzger (<a href="https://twitter.com/";>@rmetzger_</a>) + </p> + + <p><p>The Apache Flink community is pleased to announce the 1.3.0 release. Over the past 4 months, the Flink community has been working hard to resolve more than 680 issues. See the <a href="/blog/release_1.3.0-changelog.html">complete changelog</a> for more detail.</p> + +</p> + + <p><a href="/news/2017/06/01/release-1.3.0.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2017/05/16/official-docker-image.html">Introducing Docker Images for Apache Flink</a></h2> <p>16 May 2017 by Patrick Lucas (Data Artisans) and Ismaël Mejía (Talend) (<a href="https://twitter.com/";>@iemejia</a>) @@ -323,21 +338,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2016/09/05/release-1.1.2.html">Apache Flink 1.1.2 Released</a></h2> - - <p>05 Sep 2016 - </p> - - <p><p>The Apache Flink community released another bugfix version of the Apache Flink 1.1. series.</p> - -</p> - - <p><a href="/news/2016/09/05/release-1.1.2.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -370,6 +370,16 @@ <ul id="markdown-toc"> + <li><a href="/news/2020/07/06/release-1.11.0.html">Apache Flink 1.11.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 2</a></li> diff --git a/content/blog/page9/index.html b/content/blog/page9/index.html index c8497bc..406ccf6 100644 --- a/content/blog/page9/index.html +++ b/content/blog/page9/index.html @@ -196,6 +196,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2016/09/05/release-1.1.2.html">Apache Flink 1.1.2 Released</a></h2> + + <p>05 Sep 2016 + </p> + + <p><p>The Apache Flink community released another bugfix version of the Apache Flink 1.1. series.</p> + +</p> + + <p><a href="/news/2016/09/05/release-1.1.2.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2016/08/24/ff16-keynotes-panels.html">Flink Forward 2016: Announcing Schedule, Keynotes, and Panel Discussion</a></h2> <p>24 Aug 2016 @@ -327,21 +342,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2016/03/08/release-1.0.0.html">Announcing Apache Flink 1.0.0</a></h2> - - <p>08 Mar 2016 - </p> - - <p><p>The Apache Flink community is pleased to announce the availability of the 1.0.0 release. The community put significant effort into improving and extending Apache Flink since the last release, focusing on improving the experience of writing and executing data stream processing pipelines in production.</p> - -</p> - - <p><a href="/news/2016/03/08/release-1.0.0.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -374,6 +374,16 @@ <ul id="markdown-toc"> + <li><a href="/news/2020/07/06/release-1.11.0.html">Apache Flink 1.11.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 2</a></li> diff --git a/content/index.html b/content/index.html index 9c9f7dc..e5086a1 100644 --- a/content/index.html +++ b/content/index.html @@ -568,6 +568,9 @@ <dl> + <dt> <a href="/news/2020/07/06/release-1.11.0.html">Apache Flink 1.11.0 Release Announcement</a></dt> + <dd>The Apache Flink community is proud to announce the release of Flink 1.11.0! More than 200 contributors worked on over 1.3k issues to bring significant improvements to usability as well as new features to Flink users across the whole API stack. We're particularly excited about unaligned checkpoints to cope with high backpressure scenarios, a new source API that simplifies and unifies the implementation of (custom) sources, and support for Change Data Capture (CDC) and other c [...] + <dt> <a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 2</a></dt> <dd><p>In a previous post, we introduced the basics of Flink on Zeppelin and how to do Streaming ETL. In this second part of the “Flink on Zeppelin” series of posts, I will share how to perform streaming data visualization via Flink on Zeppelin and how to use Apache Flink UDFs in Zeppelin.</p> @@ -588,11 +591,6 @@ and provide a tutorial for running Streaming ETL with Flink on Zeppelin.</p> <dd><p>The Apache Flink community is happy to announce the release of Stateful Functions (StateFun) 2.1.0! This release introduces new features around state expiration and performance improvements for co-located deployments, as well as other important changes that improve the stability and testability of the project. As the community around StateFun grows, the release cycle will follow this pattern of smaller and more frequent releases to incorporate user feedback and allow for f [...] </dd> - - <dt> <a href="/news/2020/05/12/release-1.10.1.html">Apache Flink 1.10.1 Released</a></dt> - <dd><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.10 series.</p> - -</dd> </dl> diff --git a/content/zh/index.html b/content/zh/index.html index d44e4bc..77c874b 100644 --- a/content/zh/index.html +++ b/content/zh/index.html @@ -565,6 +565,9 @@ <dl> + <dt> <a href="/news/2020/07/06/release-1.11.0.html">Apache Flink 1.11.0 Release Announcement</a></dt> + <dd>The Apache Flink community is proud to announce the release of Flink 1.11.0! More than 200 contributors worked on over 1.3k issues to bring significant improvements to usability as well as new features to Flink users across the whole API stack. We're particularly excited about unaligned checkpoints to cope with high backpressure scenarios, a new source API that simplifies and unifies the implementation of (custom) sources, and support for Change Data Capture (CDC) and other c [...] + <dt> <a href="/ecosystem/2020/06/23/flink-on-zeppelin-part2.html">Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 2</a></dt> <dd><p>In a previous post, we introduced the basics of Flink on Zeppelin and how to do Streaming ETL. In this second part of the “Flink on Zeppelin” series of posts, I will share how to perform streaming data visualization via Flink on Zeppelin and how to use Apache Flink UDFs in Zeppelin.</p> @@ -585,11 +588,6 @@ and provide a tutorial for running Streaming ETL with Flink on Zeppelin.</p> <dd><p>The Apache Flink community is happy to announce the release of Stateful Functions (StateFun) 2.1.0! This release introduces new features around state expiration and performance improvements for co-located deployments, as well as other important changes that improve the stability and testability of the project. As the community around StateFun grows, the release cycle will follow this pattern of smaller and more frequent releases to incorporate user feedback and allow for f [...] </dd> - - <dt> <a href="/news/2020/05/12/release-1.10.1.html">Apache Flink 1.10.1 Released</a></dt> - <dd><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.10 series.</p> - -</dd> </dl>
