This is an automated email from the ASF dual-hosted git repository.
tison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-pulsar.git
The following commit(s) were added to refs/heads/main by this push:
new bec5feb [FLINK-26203] Add docs for the table connector (#63)
bec5feb is described below
commit bec5feb3e15c0794869c940e5fdd9e9e1d5ec592
Author: tison <[email protected]>
AuthorDate: Thu Nov 9 00:37:38 2023 +0800
[FLINK-26203] Add docs for the table connector (#63)
Signed-off-by: tison <[email protected]>
---
.../docs/connectors/datastream/pulsar.md | 3 +-
docs/content.zh/docs/connectors/table/pulsar.md | 61 ++++++++++++
docs/content/docs/connectors/datastream/pulsar.md | 3 +-
docs/content/docs/connectors/table/pulsar.md | 61 ++++++++++++
.../generated/pulsar_admin_configuration.html | 60 ------------
.../generated/pulsar_table_configuration.html | 108 +++++++++++++++++++++
6 files changed, 232 insertions(+), 64 deletions(-)
diff --git a/docs/content.zh/docs/connectors/datastream/pulsar.md
b/docs/content.zh/docs/connectors/datastream/pulsar.md
index 4625324..5aa02b0 100644
--- a/docs/content.zh/docs/connectors/datastream/pulsar.md
+++ b/docs/content.zh/docs/connectors/datastream/pulsar.md
@@ -28,7 +28,7 @@ Flink 当前提供 [Apache Pulsar](https://pulsar.apache.org) Source
和 Sink
## 添加依赖
-当前支持 Pulsar 2.10.0 及其之后的版本，建议总是将 Pulsar 升级至最新版。如果想要了解更多对于 Pulsar API
兼容性设计，可以阅读文档
[PIP-72](https://github.com/apache/pulsar/wiki/PIP-72%3A-Introduce-Pulsar-Interface-Taxonomy%3A-Audience-and-Stability-Classification)。
+当前支持 Pulsar 3.0.0 及其之后的版本，建议总是将 Pulsar 升级至最新版。如果想要了解更多对于 Pulsar API
兼容性设计，可以阅读文档
[PIP-72](https://github.com/apache/pulsar/wiki/PIP-72%3A-Introduce-Pulsar-Interface-Taxonomy%3A-Audience-and-Stability-Classification)。
{{< connector_artifact flink-connector-pulsar pulsar >}}
@@ -52,7 +52,6 @@ Pulsar Source 提供了 builder 类来构造 `PulsarSource` 实例。下面的
```java
PulsarSource<String> source = PulsarSource.builder()
.setServiceUrl(serviceUrl)
- .setAdminUrl(adminUrl)
.setStartCursor(StartCursor.earliest())
.setTopics("my-topic")
.setDeserializationSchema(new SimpleStringSchema())
diff --git a/docs/content.zh/docs/connectors/table/pulsar.md
b/docs/content.zh/docs/connectors/table/pulsar.md
new file mode 100644
index 0000000..895f48f
--- /dev/null
+++ b/docs/content.zh/docs/connectors/table/pulsar.md
@@ -0,0 +1,61 @@
+---
+title: Pulsar
+weight: 9
+type: docs
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Apache Pulsar SQL Connector
+
+{{< label "Scan Source: Unbounded" >}}
+{{< label "Scan Source: Bounded" >}}
+{{< label "Sink: Streaming Append Mode" >}}
+
+The Pulsar connector allows for reading data from and writing data into Pulsar
topics.
+
+Dependencies
+------------
+
+The Pulsar connector is not part of the binary distribution.
+See how to link with it for cluster execution [here]({{< ref
"docs/dev/configuration/overview" >}}).
+
+How to create a Pulsar table
+----------------
+
+The example below shows how to create a Pulsar table:
+
+```sql
+CREATE TABLE PulsarTable (
+ `user_id` BIGINT,
+ `item_id` BIGINT,
+ `behavior` STRING,
+ `ts` TIMESTAMP_LTZ(3) METADATA FROM 'timestamp'
+) WITH (
+ 'connector' = 'pulsar',
+ 'topics' = 'user_behavior',
+ 'service-url' = 'pulsar://my-broker.com:6650',
+)
+```
+
+Connector Options
+----------------
+
+{{< generated/pulsar_table_configuration >}}
diff --git a/docs/content/docs/connectors/datastream/pulsar.md
b/docs/content/docs/connectors/datastream/pulsar.md
index 5f991fa..54c1b47 100644
--- a/docs/content/docs/connectors/datastream/pulsar.md
+++ b/docs/content/docs/connectors/datastream/pulsar.md
@@ -28,7 +28,7 @@ Flink provides an [Apache Pulsar](https://pulsar.apache.org)
connector for readi
## Dependency
-You can use the connector with the Pulsar 2.10.0 or higher. It is recommended
to always use the latest Pulsar version.
+You can use the connector with the Pulsar 3.0.0 or higher. It is recommended
to always use the latest Pulsar version.
The details on Pulsar compatibility can be found in
[PIP-72](https://github.com/apache/pulsar/wiki/PIP-72%3A-Introduce-Pulsar-Interface-Taxonomy%3A-Audience-and-Stability-Classification).
{{< connector_artifact flink-connector-pulsar pulsar >}}
@@ -57,7 +57,6 @@ and deserializes the raw payload of the messages as strings.
```java
PulsarSource<String> source = PulsarSource.builder()
.setServiceUrl(serviceUrl)
- .setAdminUrl(adminUrl)
.setStartCursor(StartCursor.earliest())
.setTopics("my-topic")
.setDeserializationSchema(new SimpleStringSchema())
diff --git a/docs/content/docs/connectors/table/pulsar.md
b/docs/content/docs/connectors/table/pulsar.md
new file mode 100644
index 0000000..895f48f
--- /dev/null
+++ b/docs/content/docs/connectors/table/pulsar.md
@@ -0,0 +1,61 @@
+---
+title: Pulsar
+weight: 9
+type: docs
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Apache Pulsar SQL Connector
+
+{{< label "Scan Source: Unbounded" >}}
+{{< label "Scan Source: Bounded" >}}
+{{< label "Sink: Streaming Append Mode" >}}
+
+The Pulsar connector allows for reading data from and writing data into Pulsar
topics.
+
+Dependencies
+------------
+
+The Pulsar connector is not part of the binary distribution.
+See how to link with it for cluster execution [here]({{< ref
"docs/dev/configuration/overview" >}}).
+
+How to create a Pulsar table
+----------------
+
+The example below shows how to create a Pulsar table:
+
+```sql
+CREATE TABLE PulsarTable (
+ `user_id` BIGINT,
+ `item_id` BIGINT,
+ `behavior` STRING,
+ `ts` TIMESTAMP_LTZ(3) METADATA FROM 'timestamp'
+) WITH (
+ 'connector' = 'pulsar',
+ 'topics' = 'user_behavior',
+ 'service-url' = 'pulsar://my-broker.com:6650',
+)
+```
+
+Connector Options
+----------------
+
+{{< generated/pulsar_table_configuration >}}
diff --git a/docs/layouts/shortcodes/generated/pulsar_admin_configuration.html
b/docs/layouts/shortcodes/generated/pulsar_admin_configuration.html
deleted file mode 100644
index fff2f24..0000000
--- a/docs/layouts/shortcodes/generated/pulsar_admin_configuration.html
+++ /dev/null
@@ -1,60 +0,0 @@
-<table class="configuration table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 20%">Key</th>
- <th class="text-left" style="width: 15%">Default</th>
- <th class="text-left" style="width: 10%">Type</th>
- <th class="text-left" style="width: 55%">Description</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><h5>pulsar.admin.adminUrl</h5></td>
- <td style="word-wrap: break-word;">(none)</td>
- <td>String</td>
- <td>The Pulsar service HTTP URL for the admin endpoint. For
example, <code
class="highlighter-rouge">http://my-broker.example.com:8080</code>, or <code
class="highlighter-rouge">https://my-broker.example.com:8443</code> for
TLS.</td>
- </tr>
- <tr>
- <td><h5>pulsar.admin.autoCertRefreshTime</h5></td>
- <td style="word-wrap: break-word;">300000</td>
- <td>Integer</td>
- <td>The auto cert refresh time (in ms) if Pulsar admin supports
TLS authentication.</td>
- </tr>
- <tr>
- <td><h5>pulsar.admin.connectTimeout</h5></td>
- <td style="word-wrap: break-word;">60000</td>
- <td>Integer</td>
- <td>The connection time out (in ms) for the PulsarAdmin
client.</td>
- </tr>
- <tr>
- <td><h5>pulsar.admin.readTimeout</h5></td>
- <td style="word-wrap: break-word;">60000</td>
- <td>Integer</td>
- <td>The server response read timeout (in ms) for the PulsarAdmin
client for any request.</td>
- </tr>
- <tr>
- <td><h5>pulsar.admin.requestRates</h5></td>
- <td style="word-wrap: break-word;">5</td>
- <td>Integer</td>
- <td>It will add ratelimit for PulsarAdmin metadata requests,
stands for requests per second.</td>
- </tr>
- <tr>
- <td><h5>pulsar.admin.requestRetries</h5></td>
- <td style="word-wrap: break-word;">5</td>
- <td>Integer</td>
- <td>For PulsarAdmin request, it will retry until we get a success
response, fail if we exhausted retry count.</td>
- </tr>
- <tr>
- <td><h5>pulsar.admin.requestTimeout</h5></td>
- <td style="word-wrap: break-word;">300000</td>
- <td>Integer</td>
- <td>The server request timeout (in ms) for the PulsarAdmin client
for any request.</td>
- </tr>
- <tr>
- <td><h5>pulsar.admin.requestWaitMillis</h5></td>
- <td style="word-wrap: break-word;">3000</td>
- <td>Long</td>
- <td>For PulsarAdmin request, We will sleep the given time before
retrying the failed request.</td>
- </tr>
- </tbody>
-</table>
diff --git a/docs/layouts/shortcodes/generated/pulsar_table_configuration.html
b/docs/layouts/shortcodes/generated/pulsar_table_configuration.html
new file mode 100644
index 0000000..af86e89
--- /dev/null
+++ b/docs/layouts/shortcodes/generated/pulsar_table_configuration.html
@@ -0,0 +1,108 @@
+<table class="configuration table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Key</th>
+ <th class="text-left" style="width: 15%">Default</th>
+ <th class="text-left" style="width: 10%">Type</th>
+ <th class="text-left" style="width: 55%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>explicit</h5></td>
+ <td style="word-wrap: break-word;">true</td>
+ <td>Boolean</td>
+ <td>Indicate if the table is an explicit Flink table.</td>
+ </tr>
+ <tr>
+ <td><h5>key.fields</h5></td>
+ <td style="word-wrap: break-word;"></td>
+ <td>List<String></td>
+ <td>An explicit list of physical columns from the table schema
that are decoded/encoded from the key bytes of a Pulsar message. By default,
this list is empty and thus a key is undefined.</td>
+ </tr>
+ <tr>
+ <td><h5>key.format</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>Defines the format identifier for decoding/encoding key bytes
in Pulsar message. The identifier is used to discover a suitable format
factory.</td>
+ </tr>
+ <tr>
+ <td><h5>service-url</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>Service URL provider for Pulsar service.<br />To connect to
Pulsar using client libraries, you need to specify a Pulsar protocol URL.<br
/>You can assign Pulsar protocol URLs to specific clusters and use the Pulsar
scheme.<br /><ul><li>This is an example of <code
class="highlighter-rouge">localhost</code>: <code
class="highlighter-rouge">pulsar://localhost:6650</code>.</li><li>If you have
multiple brokers, the URL is as: <code
class="highlighter-rouge">pulsar://localhost: [...]
+ </tr>
+ <tr>
+ <td><h5>sink.custom-topic-router</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>(Optional) the custom topic router class URL that is used in
the [Pulsar DataStream sink
connector](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/pulsar/#pulsar-sink).
If this option is provided, the <code
class="highlighter-rouge">sink.topic-routing-mode</code> option will be
ignored.</td>
+ </tr>
+ <tr>
+ <td><h5>sink.message-delay-interval</h5></td>
+ <td style="word-wrap: break-word;">0 ms</td>
+ <td>Duration</td>
+ <td>(Optional) the message delay delivery interval that is used in
the [Pulsar DataStream sink
connector](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/pulsar/#pulsar-sink).</td>
+ </tr>
+ <tr>
+ <td><h5>sink.topic-routing-mode</h5></td>
+ <td style="word-wrap: break-word;">round-robin</td>
+ <td><p>Enum</p></td>
+ <td>(Optional) the topic routing mode. Available options are <code
class="highlighter-rouge">round-robin</code> and <code
class="highlighter-rouge">message-key-hash</code>. By default, it is set to
<code class="highlighter-rouge">round-robin</code>. If you want to use a custom
topic router, use the <code
class="highlighter-rouge">sink.custom-topic-router</code> option to determine
the partition for a particular message.<br /><br />Possible
values:<ul><li>"round-robin": The pr [...]
+ </tr>
+ <tr>
+ <td><h5>source.start.message-id</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>(Optional) Message id that is used to specify a consuming
starting point for source. Use <code class="highlighter-rouge">earliest</code>,
<code class="highlighter-rouge">latest</code> or pass in a message id
representation in <code
class="highlighter-rouge">ledgerId:entryId:partitionId</code>, such as <code
class="highlighter-rouge">12:2:-1</code>. This option takes precedence over
source.start.publish-time.</td>
+ </tr>
+ <tr>
+ <td><h5>source.start.publish-time</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>Long</td>
+ <td>(Optional) Publish timestamp that is used to specify a
starting point for the [Pulsar DataStream source
connector](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/pulsar/#pulsar-source)
to consume data. Option source.start.message-id takes precedence over this
one.</td>
+ </tr>
+ <tr>
+ <td><h5>source.stop.after-message-id</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>Optional message id used to specify a stop position but
include the given message in the consuming result for the unbounded sql source.
Pass in a message id representation in "ledgerId:entryId:partitionId", such as
"12:2:-1". </td>
+ </tr>
+ <tr>
+ <td><h5>source.stop.at-message-id</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>Optional message id used to specify a stop cursor for the
unbounded sql source. Use "never", "latest" or pass in a message id
representation in "ledgerId:entryId:partitionId", such as "12:2:-1"</td>
+ </tr>
+ <tr>
+ <td><h5>source.stop.at-publish-time</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>Long</td>
+ <td>Optional publish timestamp used to specify a stop cursor for
the unbounded sql source.</td>
+ </tr>
+ <tr>
+ <td><h5>source.subscription-name</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>The subscription name of the consumer that is used by the
runtime [Pulsar DataStream source
connector](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/pulsar/#pulsar-source).
This argument is required for constructing the consumer.</td>
+ </tr>
+ <tr>
+ <td><h5>source.subscription-type</h5></td>
+ <td style="word-wrap: break-word;">Exclusive</td>
+ <td><p>Enum</p></td>
+ <td>The [subscription
type](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/pulsar/#pulsar-subscriptions)
that is supported by the [Pulsar DataStream source
connector](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/pulsar/#pulsar-source).
Currently, only <code class="highlighter-rouge">Exclusive</code> and <code
class="highlighter-rouge">Shared</code> subscription types are supported.<br
/><br />Possible values: [...]
+ </tr>
+ <tr>
+ <td><h5>topics</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>List<String></td>
+ <td>Topic name(s) the table reads data from. It can be a single
topic name or a list of topic names separated by a semicolon symbol (<code
class="highlighter-rouge">;</code>) like <code
class="highlighter-rouge">topic-1;topic-2</code>. When a list of topics
configured, please ensure that all the topics are in the same schema as Flink
Table need a fixed schema.</td>
+ </tr>
+ <tr>
+ <td><h5>value.format</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>Defines the format identifier for decoding/encoding value
data. The identifier is used to discover a suitable format factory.</td>
+ </tr>
+ </tbody>
+</table>
