Rebuild website and update date of release 1.1.5
Project: http://git-wip-us.apache.org/repos/asf/flink-web/repo Commit: http://git-wip-us.apache.org/repos/asf/flink-web/commit/661f7648 Tree: http://git-wip-us.apache.org/repos/asf/flink-web/tree/661f7648 Diff: http://git-wip-us.apache.org/repos/asf/flink-web/diff/661f7648 Branch: refs/heads/asf-site Commit: 661f76489b9a1a1cf7b19b94e729f2cf7974a1f4 Parents: d3e398e Author: twalthr <[email protected]> Authored: Wed Mar 29 18:34:21 2017 +0200 Committer: twalthr <[email protected]> Committed: Wed Mar 29 18:34:21 2017 +0200 ---------------------------------------------------------------------- _posts/2017-03-22-release-1.1.5.md | 74 ---- _posts/2017-03-23-release-1.1.5.md | 74 ++++ content/blog/feed.xml | 261 ++++++++++--- content/blog/index.html | 33 +- content/blog/page2/index.html | 34 +- content/blog/page3/index.html | 40 +- content/blog/page4/index.html | 41 ++- content/blog/page5/index.html | 23 ++ content/index.html | 8 +- .../news/2017/03/29/table-sql-api-update.html | 364 +++++++++++++++++++ 10 files changed, 776 insertions(+), 176 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/flink-web/blob/661f7648/_posts/2017-03-22-release-1.1.5.md ---------------------------------------------------------------------- diff --git a/_posts/2017-03-22-release-1.1.5.md b/_posts/2017-03-22-release-1.1.5.md deleted file mode 100644 index 8c62cbe..0000000 --- a/_posts/2017-03-22-release-1.1.5.md +++ /dev/null @@ -1,74 +0,0 @@ ---- -layout: post -title: "Apache Flink 1.1.5 Released" -date: 2017-03-22 18:00:00 -categories: news ---- - -The Apache Flink community released the next bugfix version of the Apache Flink 1.1 series. - -This release includes critical fixes for HA recovery robustness, fault tolerance -guarantees of the Flink Kafka Connector, as well as classloading issues with the Kryo serializer. -We highly recommend all users to upgrade to Flink 1.1.5. - -```xml -<dependency> - <groupId>org.apache.flink</groupId> - <artifactId>flink-java</artifactId> - <version>1.1.5</version> -</dependency> -<dependency> - <groupId>org.apache.flink</groupId> - <artifactId>flink-streaming-java_2.10</artifactId> - <version>1.1.5</version> -</dependency> -<dependency> - <groupId>org.apache.flink</groupId> - <artifactId>flink-clients_2.10</artifactId> - <version>1.1.5</version> -</dependency> -``` - -You can find the binaries on the updated [Downloads page](http://flink.apache.org/downloads.html). - -## Release Notes - Flink - Version 1.1.5 - -### Bug -<ul> -<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5701'>FLINK-5701</a>] - FlinkKafkaProducer should check asyncException on checkpoints -</li> -<li>[<a href='https://issues.apache.org/jira/browse/FLINK-6006'>FLINK-6006</a>] - Kafka Consumer can lose state if queried partition list is incomplete on restore -</li> -<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5940'>FLINK-5940</a>] - ZooKeeperCompletedCheckpointStore cannot handle broken state handles -</li> -<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5942'>FLINK-5942</a>] - Harden ZooKeeperStateHandleStore to deal with corrupted data -</li> -<li>[<a href='https://issues.apache.org/jira/browse/FLINK-6025'>FLINK-6025</a>] - User code ClassLoader not used when KryoSerializer fallbacks to serialization for copying -</li> -<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5945'>FLINK-5945</a>] - Close function in OuterJoinOperatorBase#executeOnCollections -</li> -<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5934'>FLINK-5934</a>] - Scheduler in ExecutionGraph null if failure happens in ExecutionGraph.restoreLatestCheckpointedState -</li> -<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5771'>FLINK-5771</a>] - DelimitedInputFormat does not correctly handle multi-byte delimiters -</li> -<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5647'>FLINK-5647</a>] - Fix RocksDB Backend Cleanup -</li> -<li>[<a href='https://issues.apache.org/jira/browse/FLINK-2662'>FLINK-2662</a>] - CompilerException: "Bug: Plan generation for Unions picked a ship strategy between binary plan operators." -</li> -<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5585'>FLINK-5585</a>] - NullPointer Exception in JobManager.updateAccumulators -</li> -<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5484'>FLINK-5484</a>] - Add test for registered Kryo types -</li> -<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5518'>FLINK-5518</a>] - HadoopInputFormat throws NPE when close() is called before open() -</li> -</ul> - -### Improvement -<ul> -<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5575'>FLINK-5575</a>] - in old releases, warn users and guide them to the latest stable docs -</li> -<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5639'>FLINK-5639</a>] - Clarify License implications of RabbitMQ Connector -</li> -<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5466'>FLINK-5466</a>] - Make production environment default in gulpfile -</li> -</ul> http://git-wip-us.apache.org/repos/asf/flink-web/blob/661f7648/_posts/2017-03-23-release-1.1.5.md ---------------------------------------------------------------------- diff --git a/_posts/2017-03-23-release-1.1.5.md b/_posts/2017-03-23-release-1.1.5.md new file mode 100644 index 0000000..847f3c7 --- /dev/null +++ b/_posts/2017-03-23-release-1.1.5.md @@ -0,0 +1,74 @@ +--- +layout: post +title: "Apache Flink 1.1.5 Released" +date: 2017-03-23 18:00:00 +categories: news +--- + +The Apache Flink community released the next bugfix version of the Apache Flink 1.1 series. + +This release includes critical fixes for HA recovery robustness, fault tolerance +guarantees of the Flink Kafka Connector, as well as classloading issues with the Kryo serializer. +We highly recommend all users to upgrade to Flink 1.1.5. + +```xml +<dependency> + <groupId>org.apache.flink</groupId> + <artifactId>flink-java</artifactId> + <version>1.1.5</version> +</dependency> +<dependency> + <groupId>org.apache.flink</groupId> + <artifactId>flink-streaming-java_2.10</artifactId> + <version>1.1.5</version> +</dependency> +<dependency> + <groupId>org.apache.flink</groupId> + <artifactId>flink-clients_2.10</artifactId> + <version>1.1.5</version> +</dependency> +``` + +You can find the binaries on the updated [Downloads page](http://flink.apache.org/downloads.html). + +## Release Notes - Flink - Version 1.1.5 + +### Bug +<ul> +<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5701'>FLINK-5701</a>] - FlinkKafkaProducer should check asyncException on checkpoints +</li> +<li>[<a href='https://issues.apache.org/jira/browse/FLINK-6006'>FLINK-6006</a>] - Kafka Consumer can lose state if queried partition list is incomplete on restore +</li> +<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5940'>FLINK-5940</a>] - ZooKeeperCompletedCheckpointStore cannot handle broken state handles +</li> +<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5942'>FLINK-5942</a>] - Harden ZooKeeperStateHandleStore to deal with corrupted data +</li> +<li>[<a href='https://issues.apache.org/jira/browse/FLINK-6025'>FLINK-6025</a>] - User code ClassLoader not used when KryoSerializer fallbacks to serialization for copying +</li> +<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5945'>FLINK-5945</a>] - Close function in OuterJoinOperatorBase#executeOnCollections +</li> +<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5934'>FLINK-5934</a>] - Scheduler in ExecutionGraph null if failure happens in ExecutionGraph.restoreLatestCheckpointedState +</li> +<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5771'>FLINK-5771</a>] - DelimitedInputFormat does not correctly handle multi-byte delimiters +</li> +<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5647'>FLINK-5647</a>] - Fix RocksDB Backend Cleanup +</li> +<li>[<a href='https://issues.apache.org/jira/browse/FLINK-2662'>FLINK-2662</a>] - CompilerException: "Bug: Plan generation for Unions picked a ship strategy between binary plan operators." +</li> +<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5585'>FLINK-5585</a>] - NullPointer Exception in JobManager.updateAccumulators +</li> +<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5484'>FLINK-5484</a>] - Add test for registered Kryo types +</li> +<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5518'>FLINK-5518</a>] - HadoopInputFormat throws NPE when close() is called before open() +</li> +</ul> + +### Improvement +<ul> +<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5575'>FLINK-5575</a>] - in old releases, warn users and guide them to the latest stable docs +</li> +<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5639'>FLINK-5639</a>] - Clarify License implications of RabbitMQ Connector +</li> +<li>[<a href='https://issues.apache.org/jira/browse/FLINK-5466'>FLINK-5466</a>] - Make production environment default in gulpfile +</li> +</ul> http://git-wip-us.apache.org/repos/asf/flink-web/blob/661f7648/content/blog/feed.xml ---------------------------------------------------------------------- diff --git a/content/blog/feed.xml b/content/blog/feed.xml index 0f50aab..b677b23 100644 --- a/content/blog/feed.xml +++ b/content/blog/feed.xml @@ -7,6 +7,183 @@ <atom:link href="http://flink.apache.org/blog/feed.xml"; rel="self" type="application/rss+xml" /> <item> +<title>From Streams to Tables and Back Again: An Update on Flink's Table & SQL API</title> +<description><p>Stream processing can deliver a lot of value. Many organizations have recognized the benefit of managing large volumes of data in real-time, reacting quickly to trends, and providing customers with live services at scale. Streaming applications with well-defined business logic can deliver a competitive advantage.</p> + +<p>Flinkâs <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/datastream_api.html">DataStream</a> abstraction is a powerful API which lets you flexibly define both basic and complex streaming pipelines. Additionally, it offers low-level operations such as <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/asyncio.html">Async IO</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/process_function.html">ProcessFunctions</a>. However, many users do not need such a deep level of flexibility. They need an API which quickly solves 80% of their use cases where simple tasks can be defined using little code.</p> + +<p>To deliver the power of stream processing to a broader set of users, the Apache Flink community is developing APIs that provide simpler abstractions and more concise syntax so that users can focus on their business logic instead of advanced streaming concepts. Along with other APIs (such as <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/libs/cep.html">CEP</a> for complex event processing on streams), Flink offers a relational API that aims to unify stream and batch processing: the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/table_api.html">Table &amp; SQL API</a>, often referred to as the Table API.</p> + +<p>Recently, contributors working for companies such as Alibaba, Huawei, data Artisans, and more decided to further develop the Table API. Over the past year, the Table API has been rewritten entirely. Since Flink 1.1, its core has been based on <a href="http://calcite.apache.org/">Apache Calcite</a>, which parses SQL and optimizes all relational queries. Today, the Table API can address a wide range of use cases in both batch and stream environments with unified semantics.</p> + +<p>This blog post summarizes the current status of Flinkâs Table API and showcases some of the recently-added features in Apache Flink. Among the features presented here are the unified access to batch and streaming data, data transformation, and window operators. +The following paragraphs are not only supposed to give you a general overview of the Table API, but also to illustrate the potential of relational APIs in the future.</p> + +<p>Because the Table API is built on top of Flinkâs core APIs, <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/datastream_api.html">DataStreams</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/batch/index.html">DataSets</a> can be converted to a Table and vice-versa without much overhead. Hereafter, we show how to create tables from different sources and specify programs that can be executed locally or in a distributed setting. In this post, we will use the Scala version of the Table API, but there is also a Java version as well as a SQL API with an equivalent set of features.</p> + +<h2 id="data-transformation-and-etl">Data Transformation and ETL</h2> + +<p>A common task in every data processing pipeline is importing data from one or multiple systems, applying some transformations to it, then exporting the data to another system. The Table API can help to manage these recurring tasks. For reading data, the API provides a set of ready-to-use <code>TableSources</code> such as a <code>CsvTableSource</code> and <code>KafkaTableSource</code>, however, it also allows the implementation of custom <code>TableSources</code> that can hide configuration specifics (e.g. watermark generation) from users who are less familiar with streaming concepts.</p> + +<p>Letâs assume we have a CSV file that stores customer information. The values are delimited by a â|â-character and contain a customer identifier, name, timestamp of the last update, and preferences encoded in a comma-separated key-value string:</p> + +<div class="highlight"><pre><code>42|Bob Smith|2016-07-23 16:10:11|color=12,length=200,size=200 +</code></pre></div> + +<p>The following example illustrates how to read a CSV file and perform some data cleansing before converting it to a regular DataStream program.</p> + +<div class="highlight"><pre><code class="language-scala"><span class="c1">// set up execution environment</span> +<span class="k">val</span> <span class="n">env</span> <span class="k">=</span> <span class="nc">StreamExecutionEnvironment</span><span class="o">.</span><span class="n">getExecutionEnvironment</span> +<span class="k">val</span> <span class="n">tEnv</span> <span class="k">=</span> <span class="nc">TableEnvironment</span><span class="o">.</span><span class="n">getTableEnvironment</span><span class="o">(</span><span class="n">env</span><span class="o">)</span> + +<span class="c1">// configure table source</span> +<span class="k">val</span> <span class="n">customerSource</span> <span class="k">=</span> <span class="nc">CsvTableSource</span><span class="o">.</span><span class="n">builder</span><span class="o">()</span> + <span class="o">.</span><span class="n">path</span><span class="o">(</span><span class="s">&quot;/path/to/customer_data.csv&quot;</span><span class="o">)</span> + <span class="o">.</span><span class="n">ignoreFirstLine</span><span class="o">()</span> + <span class="o">.</span><span class="n">fieldDelimiter</span><span class="o">(</span><span class="s">&quot;|&quot;</span><span class="o">)</span> + <span class="o">.</span><span class="n">field</span><span class="o">(</span><span class="s">&quot;id&quot;</span><span class="o">,</span> <span class="nc">Types</span><span class="o">.</span><span class="nc">LONG</span><span class="o">)</span> + <span class="o">.</span><span class="n">field</span><span class="o">(</span><span class="s">&quot;name&quot;</span><span class="o">,</span> <span class="nc">Types</span><span class="o">.</span><span class="nc">STRING</span><span class="o">)</span> + <span class="o">.</span><span class="n">field</span><span class="o">(</span><span class="s">&quot;last_update&quot;</span><span class="o">,</span> <span class="nc">Types</span><span class="o">.</span><span class="nc">TIMESTAMP</span><span class="o">)</span> + <span class="o">.</span><span class="n">field</span><span class="o">(</span><span class="s">&quot;prefs&quot;</span><span class="o">,</span> <span class="nc">Types</span><span class="o">.</span><span class="nc">STRING</span><span class="o">)</span> + <span class="o">.</span><span class="n">build</span><span class="o">()</span> + +<span class="c1">// name your table source</span> +<span class="n">tEnv</span><span class="o">.</span><span class="n">registerTableSource</span><span class="o">(</span><span class="s">&quot;customers&quot;</span><span class="o">,</span> <span class="n">customerSource</span><span class="o">)</span> + +<span class="c1">// define your table program</span> +<span class="k">val</span> <span class="n">table</span> <span class="k">=</span> <span class="n">tEnv</span> + <span class="o">.</span><span class="n">scan</span><span class="o">(</span><span class="s">&quot;customers&quot;</span><span class="o">)</span> + <span class="o">.</span><span class="n">filter</span><span class="o">(</span><span class="-Symbol">&#39;name</span><span class="o">.</span><span class="n">isNotNull</span> <span class="o">&amp;&amp;</span> <span class="-Symbol">&#39;last_update</span> <span class="o">&gt;</span> <span class="s">&quot;2016-01-01 00:00:00&quot;</span><span class="o">.</span><span class="n">toTimestamp</span><span class="o">)</span> + <span class="o">.</span><span class="n">select</span><span class="o">(</span><span class="-Symbol">&#39;id</span><span class="o">,</span> <span class="-Symbol">&#39;name</span><span class="o">.</span><span class="n">lowerCase</span><span class="o">(),</span> <span class="-Symbol">&#39;prefs</span><span class="o">)</span> + +<span class="c1">// convert it to a data stream</span> +<span class="k">val</span> <span class="n">ds</span> <span class="k">=</span> <span class="n">table</span><span class="o">.</span><span class="n">toDataStream</span><span class="o">[</span><span class="kt">Row</span><span class="o">]</span> + +<span class="n">ds</span><span class="o">.</span><span class="n">print</span><span class="o">()</span> +<span class="n">env</span><span class="o">.</span><span class="n">execute</span><span class="o">()</span></code></pre></div> + +<p>The Table API comes with a large set of built-in functions that make it easy to specify business logic using a language integrated query (LINQ) syntax. In the example above, we filter out customers with invalid names and only select those that updated their preferences recently. We convert names to lowercase for normalization. For debugging purposes, we convert the table into a DataStream and print it.</p> + +<p>The <code>CsvTableSource</code> supports both batch and stream environments. If the programmer wants to execute the program above in a batch application, all he or she has to do is to replace the environment via <code>ExecutionEnvironment</code> and change the output conversion from <code>DataStream</code> to <code>DataSet</code>. The Table API program itself doesnât change.</p> + +<p>In the example, we converted the table program to a data stream of <code>Row</code> objects. However, we are not limited to row data types. The Table API supports all types from the underlying APIs such as Java and Scala Tuples, Case Classes, POJOs, or generic types that are serialized using Kryo. Letâs assume that we want to have regular object (POJO) with the following format instead of generic rows:</p> + +<div class="highlight"><pre><code class="language-scala"><span class="k">class</span> <span class="nc">Customer</span> <span class="o">{</span> + <span class="k">var</span> <span class="n">id</span><span class="k">:</span> <span class="kt">Int</span> <span class="o">=</span> <span class="k">_</span> + <span class="k">var</span> <span class="n">name</span><span class="k">:</span> <span class="kt">String</span> <span class="o">=</span> <span class="k">_</span> + <span class="k">var</span> <span class="n">update</span><span class="k">:</span> <span class="kt">Long</span> <span class="o">=</span> <span class="k">_</span> + <span class="k">var</span> <span class="n">prefs</span><span class="k">:</span> <span class="kt">java.util.Properties</span> <span class="o">=</span> <span class="k">_</span> +<span class="o">}</span></code></pre></div> +<p>We can use the following table program to convert the CSV file into Customer objects. Flink takes care of creating objects and mapping fields for us.</p> + +<div class="highlight"><pre><code class="language-scala"><span class="k">val</span> <span class="n">ds</span> <span class="k">=</span> <span class="n">tEnv</span> + <span class="o">.</span><span class="n">scan</span><span class="o">(</span><span class="s">&quot;customers&quot;</span><span class="o">)</span> + <span class="o">.</span><span class="n">select</span><span class="o">(</span><span class="-Symbol">&#39;id</span><span class="o">,</span> <span class="-Symbol">&#39;name</span><span class="o">,</span> <span class="-Symbol">&#39;last_update</span> <span class="n">as</span> <span class="-Symbol">&#39;update</span><span class="o">,</span> <span class="n">parseProperties</span><span class="o">(</span><span class="-Symbol">&#39;prefs</span><span class="o">)</span> <span class="n">as</span> <span class="-Symbol">&#39;prefs</span><span class="o">)</span> + <span class="o">.</span><span class="n">toDataStream</span><span class="o">[</span><span class="kt">Customer</span><span class="o">]</span></code></pre></div> + +<p>You might have noticed that the query above uses a function to parse the preferences field. Even though Flinkâs Table API is shipped with a large set of built-in functions, is often necessary to define custom user-defined scalar functions. In the above example we use a user-defined function <code>parseProperties</code>. The following code snippet shows how easily we can implement a scalar function.</p> + +<div class="highlight"><pre><code class="language-scala"><span class="k">object</span> <span class="nc">parseProperties</span> <span class="k">extends</span> <span class="nc">ScalarFunction</span> <span class="o">{</span> + <span class="k">def</span> <span class="n">eval</span><span class="o">(</span><span class="n">str</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">Properties</span> <span class="o">=</span> <span class="o">{</span> + <span class="k">val</span> <span class="n">props</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">Properties</span><span class="o">()</span> + <span class="n">str</span> + <span class="o">.</span><span class="n">split</span><span class="o">(</span><span class="s">&quot;,&quot;</span><span class="o">)</span> + <span class="o">.</span><span class="n">map</span><span class="o">(\</span><span class="k">_</span><span class="o">.</span><span class="n">split</span><span class="o">(</span><span class="s">&quot;=&quot;</span><span class="o">))</span> + <span class="o">.</span><span class="n">foreach</span><span class="o">(</span><span class="n">split</span> <span class="k">=&gt;</span> <span class="n">props</span><span class="o">.</span><span class="n">setProperty</span><span class="o">(</span><span class="n">split</span><span class="o">(</span><span class="mi">0</span><span class="o">),</span> <span class="n">split</span><span class="o">(</span><span class="mi">1</span><span class="o">)))</span> + <span class="n">props</span> + <span class="o">}</span> +<span class="o">}</span></code></pre></div> + +<p>Scalar functions can be used to deserialize, extract, or convert values (and more). By overwriting the <code>open()</code> method we can even have access to runtime information such as distributed cached files or metrics. Even the <code>open()</code> method is only called once during the runtimeâs <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.3/internals/task_lifecycle.html">task lifecycle</a>.</p> + +<h2 id="unified-windowing-for-static-and-streaming-data">Unified Windowing for Static and Streaming Data</h2> + +<p>Another very common task, especially when working with continuous data, is the definition of windows to split a stream into pieces of finite size, over which we can apply computations. At the moment, the Table API supports three types of windows: sliding windows, tumbling windows, and session windows (for general definitions of the different types of windows, we recommend <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/windows.html">Flinkâs documentation</a>). All three window types work on <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/event_time.html">event or processing time</a>. Session windows can be defined over time intervals, sliding and tumbling windows can be defined over time intervals or a number of rows.</p> + +<p>Letâs assume that our customer data from the example above is an event stream of updates generated whenever the customer updated his or her preferences. We assume that events come from a TableSource that has assigned timestamps and watermarks. The definition of a window happens again in a LINQ-style fashion. The following example could be used to count the updates to the preferences during one day.</p> + +<div class="highlight"><pre><code class="language-scala"><span class="n">table</span> + <span class="o">.</span><span class="n">window</span><span class="o">(</span><span class="nc">Tumble</span> <span class="n">over</span> <span class="mf">1.d</span><span class="n">ay</span> <span class="n">on</span> <span class="-Symbol">&#39;rowtime</span> <span class="n">as</span> <span class="-Symbol">&#39;w</span><span class="o">)</span> + <span class="o">.</span><span class="n">groupBy</span><span class="o">(</span><span class="-Symbol">&#39;id</span><span class="o">,</span> <span class="-Symbol">&#39;w</span><span class="o">)</span> + <span class="o">.</span><span class="n">select</span><span class="o">(</span><span class="-Symbol">&#39;id</span><span class="o">,</span> <span class="-Symbol">&#39;w</span><span class="o">.</span><span class="n">start</span> <span class="n">as</span> <span class="-Symbol">&#39;from</span><span class="o">,</span> <span class="-Symbol">&#39;w</span><span class="o">.</span><span class="n">end</span> <span class="n">as</span> <span class="-Symbol">&#39;to</span><span class="o">,</span> <span class="-Symbol">&#39;prefs</span><span class="o">.</span>&l t;span class="n">count</span> <span class="n">as</span> <span class="-Symbol">&#39;updates</span><span class="o">)</span></code></pre></div> + +<p>By using the <code>on()</code> parameter, we can specify whether the window is supposed to work on event-time or not. The Table API assumes that timestamps and watermarks are assigned correctly when using event-time. Elements with timestamps smaller than the last received watermark are dropped. Since the extraction of timestamps and generation of watermarks depends on the data source and requires some deeper knowledge of their origin, the TableSource or the upstream DataStream is usually responsible for assigning these properties.</p> + +<p>The following code shows how to define other types of windows:</p> + +<div class="highlight"><pre><code class="language-scala"><span class="c1">// using processing-time</span> +<span class="n">table</span><span class="o">.</span><span class="n">window</span><span class="o">(</span><span class="nc">Tumble</span> <span class="n">over</span> <span class="mf">100.</span><span class="n">rows</span> <span class="n">as</span> <span class="-Symbol">&#39;manyRowWindow</span><span class="o">)</span> +<span class="c1">// using event-time</span> +<span class="n">table</span><span class="o">.</span><span class="n">window</span><span class="o">(</span><span class="nc">Session</span> <span class="n">withGap</span> <span class="mf">15.</span><span class="n">minutes</span> <span class="n">on</span> <span class="-Symbol">&#39;rowtime</span> <span class="n">as</span> <span class="-Symbol">&#39;sessionWindow</span><span class="o">)</span> +<span class="n">table</span><span class="o">.</span><span class="n">window</span><span class="o">(</span><span class="nc">Slide</span> <span class="n">over</span> <span class="mf">1.d</span><span class="n">ay</span> <span class="n">every</span> <span class="mf">1.</span><span class="n">hour</span> <span class="n">on</span> <span class="-Symbol">&#39;rowtime</span> <span class="n">as</span> <span class="-Symbol">&#39;dailyWindow</span><span class="o">)</span></code></pre></div> + +<p>Since batch is just a special case of streaming (where a batch happens to have a defined start and end point), it is also possible to apply all of these windows in a batch execution environment. Without any modification of the table program itself, we can run the code on a DataSet given that we specified a column named ârowtimeâ. This is particularly interesting if we want to compute exact results from time-to-time, so that late events that are heavily out-of-order can be included in the computation.</p> + +<p>At the moment, the Table API only supports so-called âgroup windowsâ that also exist in the DataStream API. Other windows such as SQLâs OVER clause windows are in development and <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-11%3A+Table+API+Stream+Aggregations">planned for Flink 1.3</a>.</p> + +<p>In order to demonstrate the expressiveness and capabilities of the API, hereâs a snippet with a more advanced example of an exponentially decaying moving average over a sliding window of one hour which returns aggregated results every second. The table program weighs recent orders more heavily than older orders. This example is borrowed from <a href="https://calcite.apache.org/docs/stream.html#hopping-windows">Apache Calcite</a> and shows what will be possible in future Flink releases for both the Table API and SQL.</p> + +<div class="highlight"><pre><code class="language-scala"><span class="n">table</span> + <span class="o">.</span><span class="n">window</span><span class="o">(</span><span class="nc">Slide</span> <span class="n">over</span> <span class="mf">1.</span><span class="n">hour</span> <span class="n">every</span> <span class="mf">1.</span><span class="n">second</span> <span class="n">as</span> <span class="-Symbol">&#39;w</span><span class="o">)</span> + <span class="o">.</span><span class="n">groupBy</span><span class="o">(</span><span class="-Symbol">&#39;productId</span><span class="o">,</span> <span class="-Symbol">&#39;w</span><span class="o">)</span> + <span class="o">.</span><span class="n">select</span><span class="o">(</span> + <span class="-Symbol">&#39;w</span><span class="o">.</span><span class="n">end</span><span class="o">,</span> + <span class="-Symbol">&#39;productId</span><span class="o">,</span> + <span class="o">(</span><span class="-Symbol">&#39;unitPrice</span> <span class="o">*</span> <span class="o">(</span><span class="-Symbol">&#39;rowtime</span> <span class="o">-</span> <span class="-Symbol">&#39;w</span><span class="o">.</span><span class="n">start</span><span class="o">).</span><span class="n">exp</span><span class="o">()</span> <span class="o">/</span> <span class="mf">1.</span><span class="n">hour</span><span class="o">).</span><span class="n">sum</span> <span class="o">/</span> <span class="o">((</span><span class="-Symbol"& gt;&#39;rowtime</span> <span class="o">-</span> <span class="-Symbol">&#39;w</span><span class="o">.</span><span class="n">start</span><span class="o">).</span><span class="n">exp</span><span class="o">()</span> <span class="o">/</span> <span class="mf">1.</span><span class="n">hour</span><span class="o">).</span><span class="n">sum</span><span class="o">)</span></code></pre></div> + +<h2 id="user-defined-table-functions">User-defined Table Functions</h2> + +<p><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/table_api.html#user-defined-table-functions">User-defined table functions</a> were added in Flink 1.2. These can be quite useful for table columns containing non-atomic values which need to be extracted and mapped to separate fields before processing. Table functions take an arbitrary number of scalar values and allow for returning an arbitrary number of rows as output instead of a single value, similar to a flatMap function in the DataStream or DataSet API. The output of a table function can then be joined with the original row in the table by using either a left-outer join or cross join.</p> + +<p>Using the previously-mentioned customer table, letâs assume we want to produce a table that contains the color and size preferences as separate columns. The table program would look like this:</p> + +<div class="highlight"><pre><code class="language-scala"><span class="c1">// create an instance of the table function</span> +<span class="k">val</span> <span class="n">extractPrefs</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">PropertiesExtractor</span><span class="o">()</span> + +<span class="c1">// derive rows and join them with original row</span> +<span class="n">table</span> + <span class="o">.</span><span class="n">join</span><span class="o">(</span><span class="n">extractPrefs</span><span class="o">(</span><span class="-Symbol">&#39;prefs</span><span class="o">)</span> <span class="n">as</span> <span class="o">(</span><span class="-Symbol">&#39;color</span><span class="o">,</span> <span class="-Symbol">&#39;size</span><span class="o">))</span> + <span class="o">.</span><span class="n">select</span><span class="o">(</span><span class="-Symbol">&#39;id</span><span class="o">,</span> <span class="-Symbol">&#39;username</span><span class="o">,</span> <span class="-Symbol">&#39;color</span><span class="o">,</span> <span class="-Symbol">&#39;size</span><span class="o">)</span></code></pre></div> + +<p>The <code>PropertiesExtractor</code> is a user-defined table function that extracts the color and size. We are not interested in customers that havenât set these preferences and thus donât emit anything if both properties are not present in the string value. Since we are using a (cross) join in the program, customers without a result on the right side of the join will be filtered out.</p> + +<div class="highlight"><pre><code class="language-scala"><span class="k">class</span> <span class="nc">PropertiesExtractor</span> <span class="k">extends</span> <span class="nc">TableFunction</span><span class="o">[</span><span class="kt">Row</span><span class="o">]</span> <span class="o">{</span> + <span class="k">def</span> <span class="n">eval</span><span class="o">(</span><span class="n">prefs</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="o">{</span> + <span class="c1">// split string into (key, value) pairs</span> + <span class="k">val</span> <span class="n">pairs</span> <span class="k">=</span> <span class="n">prefs</span> + <span class="o">.</span><span class="n">split</span><span class="o">(</span><span class="s">&quot;,&quot;</span><span class="o">)</span> + <span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="n">kv</span> <span class="k">=&gt;</span> + <span class="k">val</span> <span class="n">split</span> <span class="k">=</span> <span class="n">kv</span><span class="o">.</span><span class="n">split</span><span class="o">(</span><span class="s">&quot;=&quot;</span><span class="o">)</span> + <span class="o">(</span><span class="n">split</span><span class="o">(</span><span class="mi">0</span><span class="o">),</span> <span class="n">split</span><span class="o">(</span><span class="mi">1</span><span class="o">))</span> + <span class="o">}</span> + + <span class="k">val</span> <span class="n">color</span> <span class="k">=</span> <span class="n">pairs</span><span class="o">.</span><span class="n">find</span><span class="o">(\</span><span class="k">_</span><span class="o">.\</span><span class="n">_1</span> <span class="o">==</span> <span class="s">&quot;color&quot;</span><span class="o">).</span><span class="n">map</span><span class="o">(\</span><span class="k">_</span><span class="o">.\</span><span class="n">_2</span><span class="o">)</span> + <span class="k">val</span> <span class="n">size</span> <span class="k">=</span> <span class="n">pairs</span><span class="o">.</span><span class="n">find</span><span class="o">(\</span><span class="k">_</span><span class="o">.\</span><span class="n">_1</span> <span class="o">==</span> <span class="s">&quot;size&quot;</span><span class="o">).</span><span class="n">map</span><span class="o">(\</span><span class="k">_</span><span class="o">.\</span><span class="n">_2</span><span class="o">)</span> + + <span class="c1">// emit a row if color and size are specified</span> + <span class="o">(</span><span class="n">color</span><span class="o">,</span> <span class="n">size</span><span class="o">)</span> <span class="k">match</span> <span class="o">{</span> + <span class="k">case</span> <span class="o">(</span><span class="nc">Some</span><span class="o">(</span><span class="n">c</span><span class="o">),</span> <span class="nc">Some</span><span class="o">(</span><span class="n">s</span><span class="o">))</span> <span class="k">=&gt;</span> <span class="n">collect</span><span class="o">(</span><span class="nc">Row</span><span class="o">.</span><span class="n">of</span><span class="o">(</span><span class="n">c</span><span class="o">,</span> <span class="n">s</span><span class="o">))& lt;/span> + <span class="k">case</span> <span class="k">_</span> <span class="k">=&gt;</span> <span class="c1">// skip</span> + <span class="o">}</span> + <span class="o">}</span> + + <span class="k">override</span> <span class="k">def</span> <span class="n">getResultType</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">RowTypeInfo</span><span class="o">(</span><span class="nc">Types</span><span class="o">.</span><span class="nc">STRING</span><span class="o">,</span> <span class="nc">Types</span><span class="o">.</span><span class="nc">STRING</span><span class="o">)</span> +<span class="o">}</span></code></pre></div> + +<h2 id="conclusion">Conclusion</h2> + +<p>There is significant interest in making streaming more accessible and easier to use. Flinkâs Table API development is happening quickly, and we believe that soon, you will be able to implement large batch or streaming pipelines using purely relational APIs or even convert existing Flink jobs to table programs. The Table API is already a very useful tool since you can work around limitations and missing features at any time by switching back-and-forth between the DataSet/DataStream abstraction to the Table abstraction.</p> + +<p>Contributions like support of Apache Hive UDFs, external catalogs, more TableSources, additional windows, and more operators will make the Table API an even more useful tool. Particularly, the upcoming introduction of Dynamic Tables, which is worth a blog post of its own, shows that even in 2017, new relational APIs open the door to a number of possibilities.</p> + +<p>Try it out, or even better, join the design discussions on the <a href="http://flink.apache.org/community.html#mailing-lists">mailing lists</a> and <a href="https://issues.apache.org/jira/browse/FLINK/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel">JIRA</a> and start contributing!</p> +</description> +<pubDate>Wed, 29 Mar 2017 14:00:00 +0200</pubDate> +<link>http://flink.apache.org/news/2017/03/29/table-sql-api-update.html</link> +<guid isPermaLink="true">/news/2017/03/29/table-sql-api-update.html</guid> +</item> + +<item> <title>Apache Flink 1.1.5 Released</title> <description><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.1 series.</p> @@ -74,7 +251,7 @@ We highly recommend all users to upgrade to Flink 1.1.5.</p> </li> </ul> </description> -<pubDate>Thu, 23 Mar 2017 02:00:00 +0800</pubDate> +<pubDate>Thu, 23 Mar 2017 19:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2017/03/23/release-1.1.5.html</link> <guid isPermaLink="true">/news/2017/03/23/release-1.1.5.html</guid> </item> @@ -348,7 +525,7 @@ If you have, for example, a flatMap() operator that keeps a running aggregate pe <li>éåå²</li> </ul> </description> -<pubDate>Mon, 06 Feb 2017 20:00:00 +0800</pubDate> +<pubDate>Mon, 06 Feb 2017 13:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2017/02/06/release-1.2.0.html</link> <guid isPermaLink="true">/news/2017/02/06/release-1.2.0.html</guid> </item> @@ -563,7 +740,7 @@ If you have, for example, a flatMap() operator that keeps a running aggregate pe </ul> </description> -<pubDate>Wed, 21 Dec 2016 17:00:00 +0800</pubDate> +<pubDate>Wed, 21 Dec 2016 10:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2016/12/21/release-1.1.4.html</link> <guid isPermaLink="true">/news/2016/12/21/release-1.1.4.html</guid> </item> @@ -757,7 +934,7 @@ enable the joining of a main, high-throughput stream with one more more inputs w <p>Lastly, weâd like to extend a sincere thank you to all of the Flink community for making 2016 a great year!</p> </description> -<pubDate>Mon, 19 Dec 2016 17:00:00 +0800</pubDate> +<pubDate>Mon, 19 Dec 2016 10:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2016/12/19/2016-year-in-review.html</link> <guid isPermaLink="true">/news/2016/12/19/2016-year-in-review.html</guid> </item> @@ -861,7 +1038,7 @@ enable the joining of a main, high-throughput stream with one more more inputs w </li> </ul> </description> -<pubDate>Wed, 12 Oct 2016 17:00:00 +0800</pubDate> +<pubDate>Wed, 12 Oct 2016 11:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2016/10/12/release-1.1.3.html</link> <guid isPermaLink="true">/news/2016/10/12/release-1.1.3.html</guid> </item> @@ -935,7 +1112,7 @@ enable the joining of a main, high-throughput stream with one more more inputs w </ul> </description> -<pubDate>Mon, 05 Sep 2016 17:00:00 +0800</pubDate> +<pubDate>Mon, 05 Sep 2016 11:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2016/09/05/release-1.1.2.html</link> <guid isPermaLink="true">/news/2016/09/05/release-1.1.2.html</guid> </item> @@ -955,7 +1132,7 @@ enable the joining of a main, high-throughput stream with one more more inputs w <p>We hope to see many community members at Flink Forward 2016. Registration is available online: <a href="http://flink-forward.org/registration/">flink-forward.org/registration</a> </p> </description> -<pubDate>Wed, 24 Aug 2016 17:00:00 +0800</pubDate> +<pubDate>Wed, 24 Aug 2016 11:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2016/08/24/ff16-keynotes-panels.html</link> <guid isPermaLink="true">/news/2016/08/24/ff16-keynotes-panels.html</guid> </item> @@ -986,7 +1163,7 @@ enable the joining of a main, high-throughput stream with one more more inputs w <p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p> </description> -<pubDate>Thu, 11 Aug 2016 17:00:00 +0800</pubDate> +<pubDate>Thu, 11 Aug 2016 11:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2016/08/11/release-1.1.1.html</link> <guid isPermaLink="true">/news/2016/08/11/release-1.1.1.html</guid> </item> @@ -1208,7 +1385,7 @@ enable the joining of a main, high-throughput stream with one more more inputs w <li>å«ä¹</li> </ul> </description> -<pubDate>Mon, 08 Aug 2016 21:00:00 +0800</pubDate> +<pubDate>Mon, 08 Aug 2016 15:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2016/08/08/release-1.1.0.html</link> <guid isPermaLink="true">/news/2016/08/08/release-1.1.0.html</guid> </item> @@ -1337,7 +1514,7 @@ enable the joining of a main, high-throughput stream with one more more inputs w <p>If this post made you curious and you want to try out Flinkâs SQL interface and the new Table API, we encourage you to do so! Simply clone the SNAPSHOT <a href="https://github.com/apache/flink/tree/master">master branch</a> and check out the <a href="https://ci.apache.org/projects/flink/flink-docs-master/apis/table.html">Table API documentation for the SNAPSHOT version</a>. Please note that the branch is under heavy development, and hence some code examples in this blog post might not work. We are looking forward to your feedback and welcome contributions.</p> </description> -<pubDate>Tue, 24 May 2016 18:00:00 +0800</pubDate> +<pubDate>Tue, 24 May 2016 12:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2016/05/24/stream-sql.html</link> <guid isPermaLink="true">/news/2016/05/24/stream-sql.html</guid> </item> @@ -1381,7 +1558,7 @@ enable the joining of a main, high-throughput stream with one more more inputs w <li>[streaming-contrib] Fix port clash in DbStateBackend tests</li> </ul> </description> -<pubDate>Wed, 11 May 2016 16:00:00 +0800</pubDate> +<pubDate>Wed, 11 May 2016 10:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2016/05/11/release-1.0.3.html</link> <guid isPermaLink="true">/news/2016/05/11/release-1.0.3.html</guid> </item> @@ -1427,7 +1604,7 @@ enable the joining of a main, high-throughput stream with one more more inputs w <li>[<a href="https://issues.apache.org/jira/browse/FLINK-3716">FLINK-3716</a>] [kafka consumer] Decreasing socket timeout so testFailOnNoBroker() will pass before JUnit timeout</li> </ul> </description> -<pubDate>Fri, 22 Apr 2016 16:00:00 +0800</pubDate> +<pubDate>Fri, 22 Apr 2016 10:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2016/04/22/release-1.0.2.html</link> <guid isPermaLink="true">/news/2016/04/22/release-1.0.2.html</guid> </item> @@ -1440,7 +1617,7 @@ enable the joining of a main, high-throughput stream with one more more inputs w <p>Read more <a href="http://flink-forward.org/">here</a>.</p> </description> -<pubDate>Thu, 14 Apr 2016 18:00:00 +0800</pubDate> +<pubDate>Thu, 14 Apr 2016 12:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2016/04/14/flink-forward-announce.html</link> <guid isPermaLink="true">/news/2016/04/14/flink-forward-announce.html</guid> </item> @@ -1633,7 +1810,7 @@ This feature will allow to prune unpromising event sequences early.</p> <p><em>Note:</em> The example code requires Flink 1.0.1 or higher.</p> </description> -<pubDate>Wed, 06 Apr 2016 18:00:00 +0800</pubDate> +<pubDate>Wed, 06 Apr 2016 12:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2016/04/06/cep-monitoring.html</link> <guid isPermaLink="true">/news/2016/04/06/cep-monitoring.html</guid> </item> @@ -1704,7 +1881,7 @@ This feature will allow to prune unpromising event sequences early.</p> </li> </ul> </description> -<pubDate>Wed, 06 Apr 2016 16:00:00 +0800</pubDate> +<pubDate>Wed, 06 Apr 2016 10:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2016/04/06/release-1.0.1.html</link> <guid isPermaLink="true">/news/2016/04/06/release-1.0.1.html</guid> </item> @@ -1831,7 +2008,7 @@ When using this backend, active state in streaming programs can grow well beyond <li>zhangminglei</li> </ul> </description> -<pubDate>Tue, 08 Mar 2016 21:00:00 +0800</pubDate> +<pubDate>Tue, 08 Mar 2016 14:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2016/03/08/release-1.0.0.html</link> <guid isPermaLink="true">/news/2016/03/08/release-1.0.0.html</guid> </item> @@ -1868,7 +2045,7 @@ When using this backend, active state in streaming programs can grow well beyond <li><a href="https://issues.apache.org/jira/browse/FLINK-3020">FLINK-3020</a>: Set number of task slots to maximum parallelism in local execution</li> </ul> </description> -<pubDate>Thu, 11 Feb 2016 16:00:00 +0800</pubDate> +<pubDate>Thu, 11 Feb 2016 09:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2016/02/11/release-0.10.2.html</link> <guid isPermaLink="true">/news/2016/02/11/release-0.10.2.html</guid> </item> @@ -2092,7 +2269,7 @@ discussion</a> on the Flink mailing lists.</p> </description> -<pubDate>Fri, 18 Dec 2015 18:00:00 +0800</pubDate> +<pubDate>Fri, 18 Dec 2015 11:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2015/12/18/a-year-in-review.html</link> <guid isPermaLink="true">/news/2015/12/18/a-year-in-review.html</guid> </item> @@ -2239,7 +2416,7 @@ While you can embed Spouts/Bolts in a Flink program and mix-and-match them with <p><sup id="fn1">1. We confess, there are three lines changed compared to a Storm project <img class="emoji" style="width:16px;height:16px;align:absmiddle" src="/img/blog/smirk.png" />âbecause the example covers local <em>and</em> remote execution. <a href="#ref1" title="Back to text.">â©</a></sup></p> </description> -<pubDate>Fri, 11 Dec 2015 18:00:00 +0800</pubDate> +<pubDate>Fri, 11 Dec 2015 11:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2015/12/11/storm-compatibility.html</link> <guid isPermaLink="true">/news/2015/12/11/storm-compatibility.html</guid> </item> @@ -2396,7 +2573,7 @@ While you can embed Spouts/Bolts in a Flink program and mix-and-match them with <p>Support for various types of windows over continuous data streams is a must-have for modern stream processors. Apache Flink is a stream processor with a very strong feature set, including a very flexible mechanism to build and evaluate windows over continuous data streams. Flink provides pre-defined window operators for common uses cases as well as a toolbox that allows to define very custom windowing logic. The Flink community will add more pre-defined window operators as we learn the requirements from our users.</p> </description> -<pubDate>Fri, 04 Dec 2015 18:00:00 +0800</pubDate> +<pubDate>Fri, 04 Dec 2015 11:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2015/12/04/Introducing-windows.html</link> <guid isPermaLink="true">/news/2015/12/04/Introducing-windows.html</guid> </item> @@ -2455,7 +2632,7 @@ While you can embed Spouts/Bolts in a Flink program and mix-and-match them with </ul> </description> -<pubDate>Fri, 27 Nov 2015 16:00:00 +0800</pubDate> +<pubDate>Fri, 27 Nov 2015 09:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2015/11/27/release-0.10.1.html</link> <guid isPermaLink="true">/news/2015/11/27/release-0.10.1.html</guid> </item> @@ -2630,7 +2807,7 @@ Also note that some methods in the DataStream API had to be renamed as part of t </ul> </description> -<pubDate>Mon, 16 Nov 2015 16:00:00 +0800</pubDate> +<pubDate>Mon, 16 Nov 2015 09:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2015/11/16/release-0.10.0.html</link> <guid isPermaLink="true">/news/2015/11/16/release-0.10.0.html</guid> </item> @@ -3521,7 +3698,7 @@ Either <code>0 + absolutePointer</code> or <code>objectRefAddr </div> </description> -<pubDate>Wed, 16 Sep 2015 16:00:00 +0800</pubDate> +<pubDate>Wed, 16 Sep 2015 10:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2015/09/16/off-heap-memory.html</link> <guid isPermaLink="true">/news/2015/09/16/off-heap-memory.html</guid> </item> @@ -3573,7 +3750,7 @@ fault tolerance, the internal runtime architecture, and others.</p> register for the conference.</p> </description> -<pubDate>Thu, 03 Sep 2015 16:00:00 +0800</pubDate> +<pubDate>Thu, 03 Sep 2015 10:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2015/09/03/flink-forward.html</link> <guid isPermaLink="true">/news/2015/09/03/flink-forward.html</guid> </item> @@ -3634,7 +3811,7 @@ for this release:</p> <li><a href="https://issues.apache.org/jira/browse/FLINK-2584">FLINK-2584</a> ASM dependency is not shaded away</li> </ul> </description> -<pubDate>Tue, 01 Sep 2015 16:00:00 +0800</pubDate> +<pubDate>Tue, 01 Sep 2015 10:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2015/09/01/release-0.9.1.html</link> <guid isPermaLink="true">/news/2015/09/01/release-0.9.1.html</guid> </item> @@ -4091,7 +4268,7 @@ tools, graph database systems and sampling techniques.</p> <h2 id="links">Links</h2> <p><a href="https://ci.apache.org/projects/flink/flink-docs-master/libs/gelly_guide.html">Gelly Documentation</a></p> </description> -<pubDate>Mon, 24 Aug 2015 00:00:00 +0800</pubDate> +<pubDate>Mon, 24 Aug 2015 00:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2015/08/24/introducing-flink-gelly.html</link> <guid isPermaLink="true">/news/2015/08/24/introducing-flink-gelly.html</guid> </item> @@ -4330,7 +4507,7 @@ tools, graph database systems and sampling techniques.</p> <p>Flink will require at least Java 7 in major releases after 0.9.0.</p> </description> -<pubDate>Wed, 24 Jun 2015 22:00:00 +0800</pubDate> +<pubDate>Wed, 24 Jun 2015 16:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2015/06/24/announcing-apache-flink-0.9.0-release.html</link> <guid isPermaLink="true">/news/2015/06/24/announcing-apache-flink-0.9.0-release.html</guid> </item> @@ -4369,7 +4546,7 @@ including Apache Flink.</p> <p>Stay tuned for a wealth of upcoming events! Two Flink talsk will be presented at <a href="http://berlinbuzzwords.de/15/sessions">Berlin Buzzwords</a>, Flink will be presented at the <a href="http://2015.hadoopsummit.org/san-jose/">Hadoop Summit in San Jose</a>. A <a href="http://www.meetup.com/Apache-Flink-Meetup/events/220557545/">training workshop on Apache Flink</a> is being organized in Berlin. Finally, <a href="http://2015.flink-forward.org/">Flink Forward</a>, the first conference to bring together the whole Flink community is taking place in Berlin in October 2015.</p> </description> -<pubDate>Thu, 14 May 2015 18:00:00 +0800</pubDate> +<pubDate>Thu, 14 May 2015 12:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2015/05/14/Community-update-April.html</link> <guid isPermaLink="true">/news/2015/05/14/Community-update-April.html</guid> </item> @@ -4560,7 +4737,7 @@ The following figure shows how two objects are compared.</p> <li>Flinkâs DBMS-style operators operate natively on binary data yielding high performance in-memory and destage gracefully to disk if necessary.</li> </ul> </description> -<pubDate>Mon, 11 May 2015 18:00:00 +0800</pubDate> +<pubDate>Mon, 11 May 2015 12:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html</link> <guid isPermaLink="true">/news/2015/05/11/Juggling-with-Bits-and-Bytes.html</guid> </item> @@ -4812,7 +4989,7 @@ Improve usability of command line interface</p> </li> </ul> </description> -<pubDate>Mon, 13 Apr 2015 18:00:00 +0800</pubDate> +<pubDate>Mon, 13 Apr 2015 12:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2015/04/13/release-0.9.0-milestone1.html</link> <guid isPermaLink="true">/news/2015/04/13/release-0.9.0-milestone1.html</guid> </item> @@ -4879,7 +5056,7 @@ limited in that it does not yet handle large state and iterative programs.</p> </description> -<pubDate>Tue, 07 Apr 2015 18:00:00 +0800</pubDate> +<pubDate>Tue, 07 Apr 2015 12:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2015/04/07/march-in-flink.html</link> <guid isPermaLink="true">/news/2015/04/07/march-in-flink.html</guid> </item> @@ -5066,7 +5243,7 @@ programs.</p> [4] <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/batch/index.html#semantic-annotations">Flink 1.0 documentation: Semantic annotations</a> <br /> [5] <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/batch/dataset_transformations.html#join-algorithm-hints">Flink 1.0 documentation: Optimizer join hints</a> <br /></p> </description> -<pubDate>Fri, 13 Mar 2015 18:00:00 +0800</pubDate> +<pubDate>Fri, 13 Mar 2015 11:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html</link> <guid isPermaLink="true">/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html</guid> </item> @@ -5182,7 +5359,7 @@ Hadoop clusters. Also, basic support for accessing secured HDFS with a standalone Flink setup is now available.</p> </description> -<pubDate>Mon, 02 Mar 2015 18:00:00 +0800</pubDate> +<pubDate>Mon, 02 Mar 2015 11:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2015/03/02/february-2015-in-flink.html</link> <guid isPermaLink="true">/news/2015/03/02/february-2015-in-flink.html</guid> </item> @@ -5827,7 +6004,7 @@ internally, fault tolerance, and performance measurements!</p> <p><a href="#top">Back to top</a></p> </description> -<pubDate>Mon, 09 Feb 2015 20:00:00 +0800</pubDate> +<pubDate>Mon, 09 Feb 2015 13:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2015/02/09/streaming-example.html</link> <guid isPermaLink="true">/news/2015/02/09/streaming-example.html</guid> </item> @@ -5876,7 +6053,7 @@ internally, fault tolerance, and performance measurements!</p> <p>The improved YARN client of Flink now allows users to deploy Flink on YARN for executing a single job. Older versions only supported a long-running YARN session. The code of the YARN client has been refactored to provide an (internal) Java API for controlling YARN clusters more easily.</p> </description> -<pubDate>Wed, 04 Feb 2015 18:00:00 +0800</pubDate> +<pubDate>Wed, 04 Feb 2015 11:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2015/02/04/january-in-flink.html</link> <guid isPermaLink="true">/news/2015/02/04/january-in-flink.html</guid> </item> @@ -5962,7 +6139,7 @@ internally, fault tolerance, and performance measurements!</p> <li>Chen Xu</li> </ul> </description> -<pubDate>Wed, 21 Jan 2015 18:00:00 +0800</pubDate> +<pubDate>Wed, 21 Jan 2015 11:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2015/01/21/release-0.8.html</link> <guid isPermaLink="true">/news/2015/01/21/release-0.8.html</guid> </item> @@ -6024,7 +6201,7 @@ Flink serialization system improved a lot over time and by now surpasses the cap <p>The community is working hard together with the Apache infra team to migrate the Flink infrastructure to a top-level project. At the same time, the Flink community is working on the Flink 0.8.0 release which should be out very soon.</p> </description> -<pubDate>Tue, 06 Jan 2015 18:00:00 +0800</pubDate> +<pubDate>Tue, 06 Jan 2015 11:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2015/01/06/december-in-flink.html</link> <guid isPermaLink="true">/news/2015/01/06/december-in-flink.html</guid> </item> @@ -6111,7 +6288,7 @@ Flink serialization system improved a lot over time and by now surpasses the cap <p>If you want to use Flinkâs Hadoop compatibility package checkout our <a href="https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/hadoop_compatibility.html">documentation</a>.</p> </description> -<pubDate>Tue, 18 Nov 2014 18:00:00 +0800</pubDate> +<pubDate>Tue, 18 Nov 2014 11:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2014/11/18/hadoop-compatibility.html</link> <guid isPermaLink="true">/news/2014/11/18/hadoop-compatibility.html</guid> </item> @@ -6183,7 +6360,7 @@ Flink serialization system improved a lot over time and by now surpasses the cap <li>Yingjun Wu</li> </ul> </description> -<pubDate>Tue, 04 Nov 2014 18:00:00 +0800</pubDate> +<pubDate>Tue, 04 Nov 2014 11:00:00 +0100</pubDate> <link>http://flink.apache.org/news/2014/11/04/release-0.7.0.html</link> <guid isPermaLink="true">/news/2014/11/04/release-0.7.0.html</guid> </item> @@ -6282,7 +6459,7 @@ properties, some algorithms)</p> <p>http://www.meetup.com/HandsOnProgrammingEvents/events/210504392/</p> </description> -<pubDate>Fri, 03 Oct 2014 18:00:00 +0800</pubDate> +<pubDate>Fri, 03 Oct 2014 12:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2014/10/03/upcoming_events.html</link> <guid isPermaLink="true">/news/2014/10/03/upcoming_events.html</guid> </item> @@ -6296,7 +6473,7 @@ of the system. We suggest all users of Flink to work with this newest version.&l <p><a href="/downloads.html">Download</a> the release today.</p> </description> -<pubDate>Fri, 26 Sep 2014 18:00:00 +0800</pubDate> +<pubDate>Fri, 26 Sep 2014 12:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2014/09/26/release-0.6.1.html</link> <guid isPermaLink="true">/news/2014/09/26/release-0.6.1.html</guid> </item> @@ -6379,7 +6556,7 @@ robust, as well as breaking API changes.</p> <li>Tobias Wiens</li> </ul> </description> -<pubDate>Tue, 26 Aug 2014 18:00:00 +0800</pubDate> +<pubDate>Tue, 26 Aug 2014 12:00:00 +0200</pubDate> <link>http://flink.apache.org/news/2014/08/26/release-0.6.html</link> <guid isPermaLink="true">/news/2014/08/26/release-0.6.html</guid> </item> http://git-wip-us.apache.org/repos/asf/flink-web/blob/661f7648/content/blog/index.html ---------------------------------------------------------------------- diff --git a/content/blog/index.html b/content/blog/index.html index c7b2173..e9f15ae 100644 --- a/content/blog/index.html +++ b/content/blog/index.html @@ -142,6 +142,17 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2017/03/29/table-sql-api-update.html">From Streams to Tables and Back Again: An Update on Flink's Table & SQL API</a></h2> + <p>29 Mar 2017 by Timo Walther (<a href="https://twitter.com/twalthr";>@twalthr</a>)</p> + + <p><p>Broadening the user base and unifying batch & streaming with relational APIs</p></p> + + <p><a href="/news/2017/03/29/table-sql-api-update.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2017/03/23/release-1.1.5.html">Apache Flink 1.1.5 Released</a></h2> <p>23 Mar 2017</p> @@ -254,18 +265,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2016/05/24/stream-sql.html">Stream Processing for Everyone with SQL and Apache Flink</a></h2> - <p>24 May 2016 by Fabian Hueske (<a href="https://twitter.com/fhueske";>@fhueske</a>)</p> - - <p><p>About six months ago, the Apache Flink community started an effort to add a SQL interface for stream data analysis. SQL is <i>the</i> standard language to access and process data. Everybody who occasionally analyzes data is familiar with SQL. Consequently, a SQL interface for stream data processing will make this technology accessible to a much wider audience. Moreover, SQL support for streaming data will also enable new use cases such as interactive and ad-hoc stream analysis and significantly simplify many applications including stream ingestion and simple transformations.</p> -<p>In this blog post, we report on the current status, architectural design, and future plans of the Apache Flink community to implement support for SQL as a language for analyzing data streams.</p></p> - - <p><a href="/news/2016/05/24/stream-sql.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -298,6 +297,16 @@ <ul id="markdown-toc"> + <li><a href="/news/2017/03/29/table-sql-api-update.html">From Streams to Tables and Back Again: An Update on Flink's Table & SQL API</a></li> + + + + + + + + + <li><a href="/news/2017/03/23/release-1.1.5.html">Apache Flink 1.1.5 Released</a></li> http://git-wip-us.apache.org/repos/asf/flink-web/blob/661f7648/content/blog/page2/index.html ---------------------------------------------------------------------- diff --git a/content/blog/page2/index.html b/content/blog/page2/index.html index 0a18f5f..c39e2ba 100644 --- a/content/blog/page2/index.html +++ b/content/blog/page2/index.html @@ -142,6 +142,18 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2016/05/24/stream-sql.html">Stream Processing for Everyone with SQL and Apache Flink</a></h2> + <p>24 May 2016 by Fabian Hueske (<a href="https://twitter.com/fhueske";>@fhueske</a>)</p> + + <p><p>About six months ago, the Apache Flink community started an effort to add a SQL interface for stream data analysis. SQL is <i>the</i> standard language to access and process data. Everybody who occasionally analyzes data is familiar with SQL. Consequently, a SQL interface for stream data processing will make this technology accessible to a much wider audience. Moreover, SQL support for streaming data will also enable new use cases such as interactive and ad-hoc stream analysis and significantly simplify many applications including stream ingestion and simple transformations.</p> +<p>In this blog post, we report on the current status, architectural design, and future plans of the Apache Flink community to implement support for SQL as a language for analyzing data streams.</p></p> + + <p><a href="/news/2016/05/24/stream-sql.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2016/05/11/release-1.0.3.html">Flink 1.0.3 Released</a></h2> <p>11 May 2016</p> @@ -252,18 +264,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2015/12/04/Introducing-windows.html">Introducing Stream Windows in Apache Flink</a></h2> - <p>04 Dec 2015 by Fabian Hueske (<a href="https://twitter.com/fhueske";>@fhueske</a>)</p> - - <p><p>The data analysis space is witnessing an evolution from batch to stream processing for many use cases. Although batch can be handled as a special case of stream processing, analyzing never-ending streaming data often requires a shift in the mindset and comes with its own terminology (for example, âwindowingâ and âat-least-onceâ/âexactly-onceâ processing). This shift and the new terminology can be quite confusing for people being new to the space of stream processing. Apache Flink is a production-ready stream processor with an easy-to-use yet very expressive API to define advanced stream analysis programs. Flink's API features very flexible window definitions on data streams which let it stand out among other open source stream processors.</p> -<p>In this blog post, we discuss the concept of windows for stream processing, present Flink's built-in windows, and explain its support for custom windowing semantics.</p></p> - - <p><a href="/news/2015/12/04/Introducing-windows.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -296,6 +296,16 @@ <ul id="markdown-toc"> + <li><a href="/news/2017/03/29/table-sql-api-update.html">From Streams to Tables and Back Again: An Update on Flink's Table & SQL API</a></li> + + + + + + + + + <li><a href="/news/2017/03/23/release-1.1.5.html">Apache Flink 1.1.5 Released</a></li> http://git-wip-us.apache.org/repos/asf/flink-web/blob/661f7648/content/blog/page3/index.html ---------------------------------------------------------------------- diff --git a/content/blog/page3/index.html b/content/blog/page3/index.html index cce9003..75ac4b3 100644 --- a/content/blog/page3/index.html +++ b/content/blog/page3/index.html @@ -142,6 +142,18 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2015/12/04/Introducing-windows.html">Introducing Stream Windows in Apache Flink</a></h2> + <p>04 Dec 2015 by Fabian Hueske (<a href="https://twitter.com/fhueske";>@fhueske</a>)</p> + + <p><p>The data analysis space is witnessing an evolution from batch to stream processing for many use cases. Although batch can be handled as a special case of stream processing, analyzing never-ending streaming data often requires a shift in the mindset and comes with its own terminology (for example, âwindowingâ and âat-least-onceâ/âexactly-onceâ processing). This shift and the new terminology can be quite confusing for people being new to the space of stream processing. Apache Flink is a production-ready stream processor with an easy-to-use yet very expressive API to define advanced stream analysis programs. Flink's API features very flexible window definitions on data streams which let it stand out among other open source stream processors.</p> +<p>In this blog post, we discuss the concept of windows for stream processing, present Flink's built-in windows, and explain its support for custom windowing semantics.</p></p> + + <p><a href="/news/2015/12/04/Introducing-windows.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2015/11/27/release-0.10.1.html">Flink 0.10.1 released</a></h2> <p>27 Nov 2015</p> @@ -261,24 +273,6 @@ vertex-centric or gather-sum-apply to Flink dataflows.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2015/04/13/release-0.9.0-milestone1.html">Announcing Flink 0.9.0-milestone1 preview release</a></h2> - <p>13 Apr 2015</p> - - <p><p>The Apache Flink community is pleased to announce the availability of -the 0.9.0-milestone-1 release. The release is a preview of the -upcoming 0.9.0 release. It contains many new features which will be -available in the upcoming 0.9 release. Interested users are encouraged -to try it out and give feedback. As the version number indicates, this -release is a preview release that contains known issues.</p> - -</p> - - <p><a href="/news/2015/04/13/release-0.9.0-milestone1.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -311,6 +305,16 @@ release is a preview release that contains known issues.</p> <ul id="markdown-toc"> + <li><a href="/news/2017/03/29/table-sql-api-update.html">From Streams to Tables and Back Again: An Update on Flink's Table & SQL API</a></li> + + + + + + + + + <li><a href="/news/2017/03/23/release-1.1.5.html">Apache Flink 1.1.5 Released</a></li> http://git-wip-us.apache.org/repos/asf/flink-web/blob/661f7648/content/blog/page4/index.html ---------------------------------------------------------------------- diff --git a/content/blog/page4/index.html b/content/blog/page4/index.html index b967101..7d66840 100644 --- a/content/blog/page4/index.html +++ b/content/blog/page4/index.html @@ -142,6 +142,24 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2015/04/13/release-0.9.0-milestone1.html">Announcing Flink 0.9.0-milestone1 preview release</a></h2> + <p>13 Apr 2015</p> + + <p><p>The Apache Flink community is pleased to announce the availability of +the 0.9.0-milestone-1 release. The release is a preview of the +upcoming 0.9.0 release. It contains many new features which will be +available in the upcoming 0.9 release. Interested users are encouraged +to try it out and give feedback. As the version number indicates, this +release is a preview release that contains known issues.</p> + +</p> + + <p><a href="/news/2015/04/13/release-0.9.0-milestone1.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2015/04/07/march-in-flink.html">March 2015 in the Flink community</a></h2> <p>07 Apr 2015</p> @@ -263,19 +281,6 @@ and offers a new API including definition of flexible windows.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2014/10/03/upcoming_events.html">Upcoming Events</a></h2> - <p>03 Oct 2014</p> - - <p><p>We are happy to announce several upcoming Flink events both in Europe and the US. Starting with a <strong>Flink hackathon in Stockholm</strong> (Oct 8-9) and a talk about Flink at the <strong>Stockholm Hadoop User Group</strong> (Oct 8). This is followed by the very first <strong>Flink Meetup in Berlin</strong> (Oct 15). In the US, there will be two Flink Meetup talks: the first one at the <strong>Pasadena Big Data User Group</strong> (Oct 29) and the second one at <strong>Silicon Valley Hands On Programming Events</strong> (Nov 4).</p> - -</p> - - <p><a href="/news/2014/10/03/upcoming_events.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -308,6 +313,16 @@ and offers a new API including definition of flexible windows.</p> <ul id="markdown-toc"> + <li><a href="/news/2017/03/29/table-sql-api-update.html">From Streams to Tables and Back Again: An Update on Flink's Table & SQL API</a></li> + + + + + + + + + <li><a href="/news/2017/03/23/release-1.1.5.html">Apache Flink 1.1.5 Released</a></li>
