This is an automated email from the ASF dual-hosted git repository. eskabetxe pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/bahir-website.git
commit f50172d55536310baf6e2f149f399a19c323a13d Author: Joao Boto <[email protected]> AuthorDate: Tue Dec 7 11:38:10 2021 +0100 Publishing from 62bce83563e9d14722ba533e33aedc0b543f3e11 --- .asf.yaml | 19 ++++++ .../docs/flink/current/documentation/index.html | 4 ++ .../flink/current/flink-streaming-kudu/index.html | 63 +++++++++++++++---- .../flink/current/flink-streaming-pinot/index.html | 73 +++++++++++++++++----- content/feed.xml | 4 +- 5 files changed, 135 insertions(+), 28 deletions(-) diff --git a/.asf.yaml b/.asf.yaml new file mode 100644 index 0000000..6067aec --- /dev/null +++ b/.asf.yaml @@ -0,0 +1,19 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +publish: + whoami: asf-site diff --git a/content/docs/flink/current/documentation/index.html b/content/docs/flink/current/documentation/index.html index b8b4e75..033f25d 100644 --- a/content/docs/flink/current/documentation/index.html +++ b/content/docs/flink/current/documentation/index.html @@ -209,10 +209,14 @@ <p><a href="../flink-streaming-influxdb">InfluxDB connector</a></p> +<p><a href="../flink-streaming-influxdb2">InfluxDB2 connector</a> <img src="/assets/themes/apache-clean/img/new-black.png" alt="" height="36px" width="36px" /></p> + <p><a href="../flink-streaming-kudu">Kudu connector</a></p> <p><a href="../flink-streaming-netty">Netty connector</a></p> +<p><a href="../flink-streaming-pinot">Pinot connector</a> <img src="/assets/themes/apache-clean/img/new-black.png" alt="" height="36px" width="36px" /></p> + <p><a href="../flink-streaming-redis">Redis connector</a></p> </div> diff --git a/content/docs/flink/current/flink-streaming-kudu/index.html b/content/docs/flink/current/flink-streaming-kudu/index.html index 760f16c..002acc3 100644 --- a/content/docs/flink/current/flink-streaming-kudu/index.html +++ b/content/docs/flink/current/flink-streaming-kudu/index.html @@ -347,18 +347,57 @@ returning a list of Kudu <a href="https://kudu.apache.org/apidocs/org/apache/kud <p>Read more about Kudu schema design in the <a href="https://kudu.apache.org/docs/schema_design.html";>Kudu docs</a>.</p> <h3 id="supported-data-types">Supported data types</h3> -<p>| Flink/SQL | Kudu | -| ————- |:————-:| -| STRING | STRING | -| BOOLEAN | BOOL | -| TINYINT | INT8 | -| SMALLINT | INT16 | -| INT | INT32 | -| BIGINT | INT64 | -| FLOAT | FLOAT | -| DOUBLE | DOUBLE | -| BYTES | BINARY | -| TIMESTAMP(3) | UNIXTIME_MICROS |</p> + +<table> + <thead> + <tr> + <th>Flink/SQL</th> + <th style="text-align: center">Kudu</th> + </tr> + </thead> + <tbody> + <tr> + <td><code class="language-plaintext highlighter-rouge">STRING</code></td> + <td style="text-align: center">STRING</td> + </tr> + <tr> + <td><code class="language-plaintext highlighter-rouge">BOOLEAN</code></td> + <td style="text-align: center">BOOL</td> + </tr> + <tr> + <td><code class="language-plaintext highlighter-rouge">TINYINT</code></td> + <td style="text-align: center">INT8</td> + </tr> + <tr> + <td><code class="language-plaintext highlighter-rouge">SMALLINT</code></td> + <td style="text-align: center">INT16</td> + </tr> + <tr> + <td><code class="language-plaintext highlighter-rouge">INT</code></td> + <td style="text-align: center">INT32</td> + </tr> + <tr> + <td><code class="language-plaintext highlighter-rouge">BIGINT</code></td> + <td style="text-align: center">INT64</td> + </tr> + <tr> + <td><code class="language-plaintext highlighter-rouge">FLOAT</code></td> + <td style="text-align: center">FLOAT</td> + </tr> + <tr> + <td><code class="language-plaintext highlighter-rouge">DOUBLE</code></td> + <td style="text-align: center">DOUBLE</td> + </tr> + <tr> + <td><code class="language-plaintext highlighter-rouge">BYTES</code></td> + <td style="text-align: center">BINARY</td> + </tr> + <tr> + <td><code class="language-plaintext highlighter-rouge">TIMESTAMP(3)</code></td> + <td style="text-align: center">UNIXTIME_MICROS</td> + </tr> + </tbody> +</table> <p>Note:</p> <ul> diff --git a/content/docs/flink/current/flink-streaming-pinot/index.html b/content/docs/flink/current/flink-streaming-pinot/index.html index 374b60d..795b708 100644 --- a/content/docs/flink/current/flink-streaming-pinot/index.html +++ b/content/docs/flink/current/flink-streaming-pinot/index.html @@ -215,6 +215,7 @@ See how to link with them for cluster execution <a href="https://ci.apache.org/p <p>The sink class is called <code class="language-plaintext highlighter-rouge">PinotSink</code>.</p> <h2 id="architecture">Architecture</h2> + <p>The Pinot sink stores elements from upstream Flink tasks in an Apache Pinot table. We support two execution modes</p> <ul> @@ -223,6 +224,7 @@ We support two execution modes</p> </ul> <h3 id="pinotsinkwriter">PinotSinkWriter</h3> + <p>Whenever the sink receives elements from upstream tasks, they are received by an instance of the PinotSinkWriter. The <code class="language-plaintext highlighter-rouge">PinotSinkWriter</code> holds a list of <code class="language-plaintext highlighter-rouge">PinotWriterSegment</code>s where each <code class="language-plaintext highlighter-rouge">PinotWriterSegment</code> is capable of storing <code class="language-plaintext highlighter-rouge">maxRowsPerSegment</code> elements. Whenever the maximum number of elements to hold is not yet reached the <code class="language-plaintext highlighter-rouge">PinotWriterSegment</code> is considered to be active. @@ -233,8 +235,9 @@ Once the maximum number of elements to hold was reached, an active <code class=" <p>Thus, there is always one active <code class="language-plaintext highlighter-rouge">PinotWriterSegment</code> that new incoming elements will go to. Over time, the list of <code class="language-plaintext highlighter-rouge">PinotWriterSegment</code> per <code class="language-plaintext highlighter-rouge">PinotSinkWriter</code> increases up to the point where a checkpoint is created.</p> -<p><strong>Checkpointing</strong><br /> -On checkpoint creation <code class="language-plaintext highlighter-rouge">PinotSinkWriter.prepareCommit</code> gets called by the Flink environment. +<p><strong>Checkpointing</strong></p> + +<p>On checkpoint creation <code class="language-plaintext highlighter-rouge">PinotSinkWriter.prepareCommit</code> gets called by the Flink environment. This triggers the creation of <code class="language-plaintext highlighter-rouge">PinotSinkCommittable</code>s where each inactive <code class="language-plaintext highlighter-rouge">PinotWriterSegment</code> creates exactly one <code class="language-plaintext highlighter-rouge">PinotSinkCommittable</code>.</p> <p><img height="250" alt="PinotSinkWriter prepareCommit" src="docs/images/PinotSinkWriter_prepareCommit.png" /></p> @@ -244,6 +247,7 @@ The file contains a list of elements in JSON format. The serialization is done v A <code class="language-plaintext highlighter-rouge">PinotSinkCommittables</code> then holds the path to the data file on the shared filesystem as well as the minimum and maximum timestamp of all contained elements (extracted via <code class="language-plaintext highlighter-rouge">EventTimeExtractor</code>).</p> <h3 id="pinotsinkglobalcommitter">PinotSinkGlobalCommitter</h3> + <p>In order to be able to follow the guidelines for Pinot segment naming, we need to include the minimum and maximum timestamp of an element in the metadata of a segment and in its name. The minimum and maximum timestamp of all elements between two checkpoints is determined at a parallelism of 1 in the <code class="language-plaintext highlighter-rouge">PinotSinkGlobalCommitter</code>. This procedure allows recovering from failure by deleting previously uploaded segments which prevents having duplicate segments in the Pinot table.</p> @@ -263,24 +267,65 @@ Moreover, the <code class="language-plaintext highlighter-rouge">PinotSinkGlobal </ul> <h2 id="delivery-guarantees">Delivery Guarantees</h2> + <p>Resulting from the above described architecture the <code class="language-plaintext highlighter-rouge">PinotSink</code> provides an at-least-once delivery guarantee. While the failure recovery mechanism ensures that duplicate segments are prevented, there might be temporary inconsistencies in the Pinot table which can result in downstream tasks receiving an element multiple times.</p> <h2 id="options">Options</h2> -<p>| Option | Description | -| ———————- | ——————————————————————————– | -| <code class="language-plaintext highlighter-rouge">pinotControllerHost</code> | Host of the Pinot controller | -| <code class="language-plaintext highlighter-rouge">pinotControllerPort</code> | Port of the Pinot controller | -| <code class="language-plaintext highlighter-rouge">tableName</code> | Target Pinot table’s name | -| <code class="language-plaintext highlighter-rouge">maxRowsPerSegment</code> | Maximum number of rows to be stored within a Pinot segment | -| <code class="language-plaintext highlighter-rouge">tempDirPrefix</code> | Prefix for temp directories used | -| <code class="language-plaintext highlighter-rouge">jsonSerializer</code> | Serializer used to convert elements to JSON | -| <code class="language-plaintext highlighter-rouge">eventTimeExtractor</code> | Defines the way event times are extracted from received objects | -| <code class="language-plaintext highlighter-rouge">segmentNameGenerator</code> | Pinot segment name generator | -| <code class="language-plaintext highlighter-rouge">fsAdapter</code> | Filesystem adapter used to save files for sharing files across nodes | -| <code class="language-plaintext highlighter-rouge">numCommitThreads</code> | Number of threads used in the <code class="language-plaintext highlighter-rouge">PinotSinkGlobalCommitter</code> for committing segments |</p> + +<table> + <thead> + <tr> + <th>Option</th> + <th>Description</th> + </tr> + </thead> + <tbody> + <tr> + <td><code class="language-plaintext highlighter-rouge">pinotControllerHost</code></td> + <td>Host of the Pinot controller</td> + </tr> + <tr> + <td><code class="language-plaintext highlighter-rouge">pinotControllerPort</code></td> + <td>Port of the Pinot controller</td> + </tr> + <tr> + <td><code class="language-plaintext highlighter-rouge">tableName</code></td> + <td>Target Pinot table’s name</td> + </tr> + <tr> + <td><code class="language-plaintext highlighter-rouge">maxRowsPerSegment</code></td> + <td>Maximum number of rows to be stored within a Pinot segment</td> + </tr> + <tr> + <td><code class="language-plaintext highlighter-rouge">tempDirPrefix</code></td> + <td>Prefix for temp directories used</td> + </tr> + <tr> + <td><code class="language-plaintext highlighter-rouge">jsonSerializer</code></td> + <td>Serializer used to convert elements to JSON</td> + </tr> + <tr> + <td><code class="language-plaintext highlighter-rouge">eventTimeExtractor</code></td> + <td>Defines the way event times are extracted from received objects</td> + </tr> + <tr> + <td><code class="language-plaintext highlighter-rouge">segmentNameGenerator</code></td> + <td>Pinot segment name generator</td> + </tr> + <tr> + <td><code class="language-plaintext highlighter-rouge">fsAdapter</code></td> + <td>Filesystem adapter used to save files for sharing files across nodes</td> + </tr> + <tr> + <td><code class="language-plaintext highlighter-rouge">numCommitThreads</code></td> + <td>Number of threads used in the <code class="language-plaintext highlighter-rouge">PinotSinkGlobalCommitter</code> for committing segments</td> + </tr> + </tbody> +</table> <h2 id="usage">Usage</h2> + <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">StreamExecutionEnvironment</span> <span class="n">env</span> <span class="o">=</span> <span class="o">...</span> <span class="c1">// Checkpointing needs to be enabled when executing in STREAMING mode</span> <span class="n">env</span><span class="o">.</span><span class="na">enableCheckpointing</span><span class="o">(</span><span class="kt">long</span> <span class="n">interval</span><span class="o">);</span> diff --git a/content/feed.xml b/content/feed.xml index 77ed035..f422ad7 100644 --- a/content/feed.xml +++ b/content/feed.xml @@ -6,8 +6,8 @@ </description> <link>http://bahir.apache.org/</link> <atom:link href="http://bahir.apache.org/feed.xml"; rel="self" type="application/rss+xml"/> - <pubDate>Tue, 07 Dec 2021 09:18:03 +0100</pubDate> - <lastBuildDate>Tue, 07 Dec 2021 09:18:03 +0100</lastBuildDate> + <pubDate>Tue, 07 Dec 2021 11:36:34 +0100</pubDate> + <lastBuildDate>Tue, 07 Dec 2021 11:36:34 +0100</lastBuildDate> <generator>Jekyll v3.9.0</generator> </channel>
