sjwiesman commented on a change in pull request #8903: [FLINK-12747][docs] Getting Started - Table API Example Walkthrough URL: https://github.com/apache/flink/pull/8903#discussion_r306895655
########## File path: docs/getting-started/walkthroughs/table_api.md ########## @@ -0,0 +1,445 @@ +--- +title: "Table API" +nav-id: tableapiwalkthrough +nav-title: 'Table API' +nav-parent_id: walkthroughs +nav-pos: 1 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +Apache Flink offers a Table API as a unified, relational API for batch and stream processing, i.e., queries are executed with the same semantics on unbounded, real-time streams or bounded, recorded streams and produce the same results. +The Table API in Flink is commonly used to ease the definition of data analytics, data pipelining, and ETL applications. + +* This will be replaced by the TOC +{:toc} + +## What Will You Be Building? + +In this tutorial, you'll learn how to build a continuous ETL pipeline for tracking financial transactions by account over time. +You will start by building your report as a nightly batch job, and then migrate to a streaming pipeline. + +## Prerequisites + +This walkthrough assumes that you have some familiarity with Java or Scala, but you should be able to follow along even if you're coming from a different programming language. +It also assumes that you're familiar with basic relational concepts such as `SELECT` and `GROUP BY` clauses. + +## Help, I’m Stuck! + +If you get stuck, check out the [community support resources](https://flink.apache.org/community.html). +In particular, Apache Flink's [user mailing list](https://flink.apache.org/community.html#mailing-lists) is consistently ranked as one of the most active of any Apache project and a great way to get help quickly. + +## How To Follow Along + +If you want to follow along, you will require a computer with: + +* Java 8 +* Maven + +There is also a provided Flink Maven Archetype to create a skeleton project with all the necessary dependencies quickly. + +<div class="codetabs" markdown="1"> +<div data-lang="java" markdown="1"> +{% highlight bash %} +$ mvn archetype:generate \ + -DarchetypeGroupId=org.apache.flink \ + -DarchetypeArtifactId=flink-walkthrough-table-java \{% unless site.is_stable %} + -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %} + -DarchetypeVersion={{ site.version }} \ + -DgroupId=spend-report \ + -DartifactId=spend-report \ + -Dversion=0.1 \ + -Dpackage=spendreport \ + -DinteractiveMode=false +{% endhighlight %} +</div> +<div data-lang="scala" markdown="1"> +{% highlight bash %} +$ mvn archetype:generate \ + -DarchetypeGroupId=org.apache.flink \ + -DarchetypeArtifactId=flink-walkthrough-table-scala \{% unless site.is_stable %} + -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %} + -DarchetypeVersion={{ site.version }} \ + -DgroupId=spend-report \ + -DartifactId=spend-report \ + -Dversion=0.1 \ + -Dpackage=spendreport \ + -DinteractiveMode=false +{% endhighlight %} +</div> +</div> + +{% unless site.is_stable %} +<p style="border-radius: 5px; padding: 5px" class="bg-danger"> + <b>Note</b>: For Maven 3.0 or higher, it is no longer possible to specify the repository (-DarchetypeCatalog) via the commandline. If you wish to use the snapshot repository, you need to add a repository entry to your settings.xml. For details about this change, please refer to <a href="http://maven.apache.org/archetype/maven-archetype-plugin/archetype-repository.html";>Maven official document</a> +</p> +{% endunless %} + +You can edit the `groupId`, `artifactId` and `package` if you like. With the above parameters, +Maven will create a project with all the dependencies to complete this tutorial. +After importing the project into your editor, you will see a file following code. + +<div class="codetabs" markdown="1"> +<div data-lang="java" markdown="1"> +{% highlight java %} +ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); +BatchTableEnvironment tEnv = BatchTableEnvironment.create(env); + +tEnv.registerTableSource("transactions", new BoundedTransactionTableSource()); +tEnv.registerTableSink("spend_report", new SpendReportTableSink()); +tEnv.registerFunction("truncateDateToHour", new TruncateDateToHour()); + +tEnv + .scan("transactions") + .insertInto("spend_report"); + +env.execute("Spend Report"); +{% endhighlight %} +</div> + +<div data-lang="scala" markdown="1"> +{% highlight scala %} +val env = ExecutionEnvironment.getExecutionEnvironment +val tEnv = BatchTableEnvironment.create(env) + +tEnv.registerTableSource("transactions", new BoundedTransactionTableSource) +tEnv.registerTableSink("spend_report", new SpendReportTableSink) + +val truncateDateToHour = new TruncateDateToHour + +tEnv + .scan("transactions") + .insertInto("spend_report") + +env.execute("Spend Report") +{% endhighlight %} +</div> +</div> + +Let's break down this code by component. + +## Breaking Down The Code + +#### The Execution Environment + +The first two lines set up your `ExecutionEnvironment`. +The execution environment is how you can set properties for your Job, specify whether you are writing a batch or streaming application, and create your sources. +This walkthrough begins with the batch environment since we are building a periodic batch report. +It is then wrapped in a `BatchTableEnvironment` to have full access to the Table API. + +<div class="codetabs" markdown="1"> +<div data-lang="java" markdown="1"> +{% highlight java %} +ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); +BatchTableEnvironment tEnv = BatchTableEnvironment.create(env); +{% endhighlight %} +</div> + +<div data-lang="scala" markdown="1"> +{% highlight scala %} +val env = ExecutionEnvironment.getExecutionEnvironment +val tEnv = BatchTableEnvironment.create(env) +{% endhighlight %} +</div> +</div> + + +#### Registering Tables + +Next, tables are registered in the execution environment that you can use to connect to external systems for reading and writing both batch and streaming data. +A table source provides access to data stored in external systems; such as a database, key-value store, message queue, or file system. +A table sink emits a table to an external storage system. +Depending on the type of source and sink, they support different formats such as CSV, JSON, Avro, or Parquet. + +<div class="codetabs" markdown="1"> +<div data-lang="java" markdown="1"> +{% highlight java %} +tEnv.registerTableSource("transactions", new BoundedTransactionTableSource()); +tEnv.registerTableSink("spend_report", new SpendReportTableSink()); +{% endhighlight %} +</div> + +<div data-lang="scala" markdown="1"> +{% highlight scala %} +tEnv.registerTableSource("transactions", new BoundedTransactionTableSource) +tEnv.registerTableSink("spend_report", new SpendReportTableSink) +{% endhighlight %} +</div> +</div> + +Two tables are registered, a transaction input table, and a spend report output table. Review comment: Actually this should be a semicolon ";" but good catch ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
