This is an automated email from the ASF dual-hosted git repository.
lzljs3620320 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink-table-store.git
The following commit(s) were added to refs/heads/master by this push:
new 84e91308 [FLINK-28456] Document streaming query page
84e91308 is described below
commit 84e9130827c09bf5338978b8f92bed4c6220b204
Author: Jingsong Lee <[email protected]>
AuthorDate: Mon Jul 11 17:07:54 2022 +0800
[FLINK-28456] Document streaming query page
This closes #207
---
docs/content/docs/development/configuration.md | 2 +-
docs/content/docs/development/overview.md | 27 +-----
docs/content/docs/development/query-table.md | 66 ++++++++-----
docs/content/docs/development/rescale-bucket.md | 2 +-
docs/content/docs/development/streaming-query.md | 115 +++++++++++++++++++++++
docs/content/docs/development/write-table.md | 2 +-
6 files changed, 162 insertions(+), 52 deletions(-)
diff --git a/docs/content/docs/development/configuration.md
b/docs/content/docs/development/configuration.md
index aa0871d3..5d15dc35 100644
--- a/docs/content/docs/development/configuration.md
+++ b/docs/content/docs/development/configuration.md
@@ -1,6 +1,6 @@
---
title: "Configuration"
-weight: 6
+weight: 7
type: docs
aliases:
- /development/configuration.html
diff --git a/docs/content/docs/development/overview.md
b/docs/content/docs/development/overview.md
index 925844cc..6a317190 100644
--- a/docs/content/docs/development/overview.md
+++ b/docs/content/docs/development/overview.md
@@ -37,7 +37,7 @@ As shown in the architecture above:
**Read/Write:** Table Store supports a versatile way to read/write data and
perform OLAP queries.
- For reads, it supports consuming data <1> from historical snapshots (in
batch mode), <2>from the
- latest offset (in continuous mode), or <3> reading incremental snapshots in
a hybrid way.
+ latest offset (in streaming mode), or <3> reading incremental snapshots in a
hybrid way.
- For writes, it supports streaming synchronization from the changelog of
databases (CDC) or batch
insert/overwrite from offline data.
@@ -70,7 +70,7 @@ The steps to set up are:
- Setting the HADOOP_CLASSPATH environment variable or copy the
[Pre-bundled Hadoop Jar](https://flink.apache.org/downloads.html) to
`flink/lib`.
-## Unified Table
+## Unified Storage
There are three types of connectors in Flink SQL.
- Message queue, such as Apache Kafka, it is used in both source and
@@ -89,26 +89,3 @@ does not differ from the traditional database:
- In Flink `streaming` execution mode, it acts like a message queue.
Query it acts like querying a stream changelog from a message queue
where historical data never expires.
-
-Different `log.scan` mode will result in different consuming behavior under
streaming mode.
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 20%">Scan Mode</th>
- <th class="text-center" style="width: 5%">Default</th>
- <th class="text-center" style="width: 60%">Description</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><h5>FULL</h5></td>
- <td>Yes</td>
- <td>FULL scan mode performs a hybrid reading with a snapshot scan and
the continuous incremental scan.</td>
- </tr>
- <tr>
- <td><h5>LATEST</h5></td>
- <td>No</td>
- <td>LATEST scan mode only reads incremental data from the latest
offset.</td>
- </tr>
- </tbody>
-</table>
diff --git a/docs/content/docs/development/query-table.md
b/docs/content/docs/development/query-table.md
index 9805b7ba..520e1d4a 100644
--- a/docs/content/docs/development/query-table.md
+++ b/docs/content/docs/development/query-table.md
@@ -26,49 +26,67 @@ under the License.
# Query Table
-The Table Store is streaming batch unified, you can read full
-and incremental data depending on the runtime execution mode:
+You can directly SELECT the table in batch runtime mode of Flink SQL.
```sql
-- Batch mode, read latest snapshot
SET 'execution.runtime-mode' = 'batch';
SELECT * FROM MyTable;
+```
--- Streaming mode, read incremental snapshot, read the snapshot first, then
read the incremental
-SET 'execution.runtime-mode' = 'streaming';
-SELECT * FROM MyTable;
+## Query Engines
--- Streaming mode, read latest incremental
-SET 'execution.runtime-mode' = 'streaming';
-SELECT * FROM MyTable /*+ OPTIONS ('log.scan'='latest') */;
-```
+Table Store not only supports Flink SQL queries natively but also provides
+queries from other popular engines. See [Engines]({{< ref
"docs/engines/overview" >}})
## Query Optimization
It is highly recommended to specify partition and primary key filters
along with the query, which will speed up the data skipping of the query.
-along with the query, which will speed up the data skipping of the query.
-Supported filter functions are:
+The filter functions that can accelerate data skipping are:
- `=`
-- `<>`
- `<`
- `<=`
- `>`
- `>=`
-- `in`
-- starts with `like`
+- `IN (...)`
+- `LIKE 'abc%'`
+- `IS NULL`
+
+Table Store will sort the data by primary key, which speeds up the point
queries
+and range queries. When using a composite primary key, it is best for the query
+filters to form a [leftmost
prefix](https://dev.mysql.com/doc/refman/5.7/en/multiple-column-indexes.html)
+of the primary key for good acceleration.
+
+Suppose that a table has the following specification:
+
+```sql
+CREATE TABLE orders (
+ catalog_id BIGINT,
+ order_id BIGINT,
+ .....,
+ PRIMARY KEY (catalog_id, order_id) NOT ENFORCED -- composite primary key
+)
+```
+
+The query obtains a good acceleration by specifying a range filter for
+the leftmost prefix of the primary key.
+
+```sql
+SELECT * FROM orders WHERE catalog_id=1025;
-## Real-time Streaming Consumption
+SELECT * FROM orders WHERE catalog_id=1025 AND order_id=29495;
-By default, data is only visible after the checkpoint, which means
-that the streaming reading has transactional consistency.
+SELECT * FROM orders
+ WHERE catalog_id=1025
+ AND order_id>2035 AND order_id<6000;
+```
+
+However, the following filter cannot accelerate the query well.
-Immediate data visibility is configured via
-`log.consistency` = `eventual`.
+```sql
+SELECT * FROM orders WHERE order_id=29495;
-Due to the tradeoff between data freshness and completeness, immediate data
visibility is barely
-accomplished under exactly-once semantics. Nevertheless, users can relax the
constraint to use
-at-least-once mode to achieve it. Note that records may be sent to downstream
jobs ahead of the committing
-(since no barrier alignment is required), which may lead to duplicate data
during job failover. As a result,
-users may need to manually de-duplicate data to achieve final consistency.
+SELECT * FROM orders WHERE catalog_id=1025 OR order_id=29495;
+```
diff --git a/docs/content/docs/development/rescale-bucket.md
b/docs/content/docs/development/rescale-bucket.md
index 5ef95620..9cdf83e5 100644
--- a/docs/content/docs/development/rescale-bucket.md
+++ b/docs/content/docs/development/rescale-bucket.md
@@ -1,6 +1,6 @@
---
title: "Rescale Bucket"
-weight: 5
+weight: 6
type: docs
aliases:
- /development/rescale-bucket.html
diff --git a/docs/content/docs/development/streaming-query.md
b/docs/content/docs/development/streaming-query.md
new file mode 100644
index 00000000..b3e25b1e
--- /dev/null
+++ b/docs/content/docs/development/streaming-query.md
@@ -0,0 +1,115 @@
+---
+title: "Streaming Query"
+weight: 5
+type: docs
+aliases:
+- /development/streaming-query.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Streaming Query
+
+Currently, Table Store only supports Flink streaming queries.
+
+The Table Store is streaming batch unified, you can read full
+and incremental data depending on the runtime execution mode:
+
+```sql
+-- Batch mode, read latest snapshot
+SET 'execution.runtime-mode' = 'batch';
+SELECT * FROM MyTable;
+
+-- Streaming mode, streaming reading, read incremental snapshot, read the
snapshot first, then read the incremental
+SET 'execution.runtime-mode' = 'streaming';
+SELECT * FROM MyTable;
+
+-- Streaming mode, streaming reading, read latest incremental
+SET 'execution.runtime-mode' = 'streaming';
+SELECT * FROM MyTable /*+ OPTIONS ('log.scan'='latest') */;
+```
+
+Different `log.scan` mode will result in different consuming behavior under
streaming mode.
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Scan Mode</th>
+ <th class="text-center" style="width: 5%">Default</th>
+ <th class="text-center" style="width: 60%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>FULL</h5></td>
+ <td>Yes</td>
+ <td>FULL scan mode performs a hybrid reading with a snapshot scan and
the streaming incremental scan.</td>
+ </tr>
+ <tr>
+ <td><h5>LATEST</h5></td>
+ <td>No</td>
+ <td>LATEST scan mode only reads incremental data from the latest
offset.</td>
+ </tr>
+ </tbody>
+</table>
+
+## Streaming Query on Files
+
+You can choose to consume incremental changes directly from the lake store
files under
+streaming mode. This mode has a lower cost compared to Kafka but has a higher
latency,
+depending on the checkpoint interval of the writing job.
+
+By default, the downstream streaming consumption is disordered (ordered within
the key)
+stream of upsert data. If you expect an ordered CDC data stream, you can
configure it
+as follows (recommended):
+
+```sql
+CREATE TABLE T (...)
+WITH (
+ 'changelog-file' = 'true',
+ 'log.changelog-mode' = 'all'
+)
+```
+
+## Streaming Query on Kafka
+
+For a table configuring a log system like Kafka, data will be double written
to the file
+storage and the Kafka topic under streaming mode. For queries, there will be
hybrid reads
+will from incremental snapshots.
+
+```sql
+CREATE TABLE T (...)
+WITH (
+ 'log.system' = 'kafka',
+ 'kafka.bootstrap.servers' = '...',
+ 'kafka.topic' = '...'
+)
+```
+The partition of the Kafka topic needs to be the same as the number of buckets.
+
+By default, data is only visible after the checkpoint, which means
+that the streaming reading has transactional consistency.
+
+Immediate data visibility is configured via
+`log.consistency` = `eventual`.
+
+Due to the tradeoff between data freshness and completeness, immediate data
visibility is barely
+accomplished under exactly-once semantics. Nevertheless, users can relax the
constraint to use
+at-least-once mode to achieve it. Note that records may be sent to downstream
jobs ahead of the committing
+(since no barrier alignment is required), which may lead to duplicate data
during job failover. As a result,
+users may need to manually de-duplicate data to achieve final consistency.
diff --git a/docs/content/docs/development/write-table.md
b/docs/content/docs/development/write-table.md
index 54d79914..883c49a9 100644
--- a/docs/content/docs/development/write-table.md
+++ b/docs/content/docs/development/write-table.md
@@ -109,7 +109,7 @@ Please note that too short retain time or too small retain
number may result in:
- Batch query cannot find the file. For example, the table is relatively large
and
the batch query takes 10 minutes to read, but the snapshot from 10 minutes
ago
expires, at which point the batch query will read a deleted snapshot.
-- Continuous reading jobs on FileStore (Without Log System) fail to restart.
At the
+- Streaming reading jobs on FileStore (Without Log System) fail to restart. At
the
time of the job failover, the snapshot it recorded has expired.
## Performance
