This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new 965273fac [INLONG-3606] Add more guide for configure sort/manager/k8s
(#341)
965273fac is described below
commit 965273fac288b7ad5ea79bc8c108990e3f40ded3
Author: dockerzhang <[email protected]>
AuthorDate: Mon Apr 11 15:04:09 2022 +0800
[INLONG-3606] Add more guide for configure sort/manager/k8s (#341)
---
docs/deployment/k8s.md | 40 +++++++++++++++++++++
docs/modules/manager/quick_start.md | 13 ++++++-
docs/modules/sort/quick_start.md | 3 +-
.../current/deployment/k8s.md | 41 ++++++++++++++++++++++
.../current/modules/manager/quick_start.md | 12 ++++++-
.../current/modules/sort/quick_start.md | 3 +-
package.json | 8 ++---
7 files changed, 112 insertions(+), 8 deletions(-)
diff --git a/docs/deployment/k8s.md b/docs/deployment/k8s.md
index 4d0cda545..0e0d92d49 100644
--- a/docs/deployment/k8s.md
+++ b/docs/deployment/k8s.md
@@ -18,8 +18,48 @@ helm upgrade inlong --install -n inlong ./
```
## Configuration
+The configuration file is **values.yaml**, and the following tables lists the
configurable parameters of InLong and their default values.
+| Parameter
| Default |
Description
|
+|:--------------------------------------------------------------------------------:|:----------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------:|
+| `timezone`
| `Asia/Shanghai` |
World time and date for cities in all time zones
|
+| `images.pullPolicy`
| `IfNotPresent` | Image
pull policy. One of `Always`, `Never`, `IfNotPresent`
|
+| `images.<component>.repository`
| |
Docker image repository for the component
|
+| `images.<component>.tag`
| `latest` |
Docker image tag for the component
|
+| `<component>.component`
| |
Component name
|
+| `<component>.replicaCount`
| `1` |
Replicas is the desired number of replicas of a given Template
|
+| `<component>.podManagementPolicy`
| `OrderedReady` | PodManagementPolicy controls how pods
are created during initial scale up, when replacing pods on nodes, or when
scaling down |
+| `<component>.annotations`
| `{}` | The `annotations`
field can be used to attach arbitrary non-identifying metadata to objects
|
+| `<component>.tolerations`
| `[]` | Tolerations are applied to pods,
and allow (but do not require) the pods to schedule onto nodes with matching
taints |
+| `<component>.nodeSelector`
| `{}` | You can add the `nodeSelector` field
to your Pod specification and specify the node labels you want the target node
to have |
+| `<component>.affinity`
| `{}` | Node affinity is conceptually similar to
nodeSelector, allowing you to constrain which nodes your Pod can be scheduled
on based on node labels |
+| `<component>.terminationGracePeriodSeconds`
| `30` | Optional
duration in seconds the pod needs to terminate gracefully
|
+| `<component>.resources`
| `{}` |
Optionally specify how much of each resource a container needs
|
+| `<component>.port(s)`
| |
The port(s) for each component service
|
+| `<component>.env`
| `{}` |
Environment variables for each component container
|
+| <code>\<component\>.probe.\<liveness|readiness\>.enabled</code>
| `true` |
Turn on and off liveness or readiness probe
|
+|
<code>\<component\>.probe.\<liveness|readiness\>.failureThreshold</code>
| `10` |
Minimum consecutive successes for the probe
|
+|
<code>\<component\>.probe.\<liveness|readiness\>.initialDelaySeconds</code>
| `10` |
Delay before the probe is initiated
|
+|
<code>\<component\>.probe.\<liveness|readiness\>.periodSeconds</code> |
`30` |
How often to perform the probe
|
+| `<component>.volumes.name`
| |
Volume name
|
+| `<component>.volumes.size`
| `10Gi` |
Volume size
|
+| `<component>.service.annotations`
| `{}` | The
`annotations` field may need to be set when service.type is `LoadBalancer`
|
+| `<component>.service.type`
| `ClusterIP` | The `type` field determines how the
service is exposed. Valid options are `ClusterIP`, `NodePort`, `LoadBalancer`
and `ExternalName` |
+| `<component>.service.clusterIP`
| `nil` | ClusterIP is the IP
address of the service and is usually assigned randomly by the master
|
+| `<component>.service.nodePort`
| `nil` | NodePort is the port on
each node on which this service is exposed when service type is `NodePort`
|
+| `<component>.service.loadBalancerIP`
| `nil` | LoadBalancer will get
created with the IP specified in this field when service type is `LoadBalancer`
|
+| `<component>.service.externalName`
| `nil` | ExternalName is the external reference that kubedns or
equivalent will return as a CNAME record for this service, requires service
type to be `ExternalName` |
+| `<component>.service.externalIPs`
| `[]` | ExternalIPs is a list of IP
addresses for which nodes in the cluster will also accept traffic for this
service |
+| `external.mysql.enabled`
| `false` | If not exists
external MySQL, InLong will use the internal MySQL by default
|
+| `external.mysql.hostname`
| `localhost` |
External MySQL hostname
|
+| `external.mysql.port`
| `3306` |
External MySQL port
|
+| `external.mysql.username`
| `root` |
External MySQL username
|
+| `external.mysql.password`
| `password` |
External MySQL password
|
+| `external.pulsar.enabled`
| `false` | If not exists
external Pulsar, InLong will use the internal TubeMQ by default
|
+| `external.pulsar.serviceUrl`
| `localhost:6650` |
External Pulsar service URL
|
+| `external.pulsar.adminUrl`
| `localhost:8080` |
External Pulsar admin URL
|
+> The components include `agent`, `audit`, `dashboard`, `dataproxy`,
`manager`, `tubemq-manager`, `tubemq-master`, `tubemq-broker`, `zookeeper` and
`mysql`.
## Uninstall
diff --git a/docs/modules/manager/quick_start.md
b/docs/modules/manager/quick_start.md
index 1c29965e3..edc932809 100644
--- a/docs/modules/manager/quick_start.md
+++ b/docs/modules/manager/quick_start.md
@@ -36,7 +36,18 @@ spring.datasource.druid.username=root
spring.datasource.druid.password=inlong
```
-## 启动
+## Flink Plugin
+InLong support to start a Sort task by Manager, you need to configure a Flink
environment in the `plugins/flink-sort-plugin.properties`.
+```properties
+# Flink host split by coma if more than one host, such as 'host1,host2'
+flink.rest.address=127.0.0.1
+# Flink port
+flink.rest.port=8081
+# Flink jobmanager port
+flink.jobmanager.port=6123
+```
+
+## Start
```shell
bash +x bin/startup.sh
```
diff --git a/docs/modules/sort/quick_start.md b/docs/modules/sort/quick_start.md
index 32d5838c3..3ebe017e0 100644
--- a/docs/modules/sort/quick_start.md
+++ b/docs/modules/sort/quick_start.md
@@ -22,7 +22,7 @@ Example：
./bin/flink run -c org.apache.inlong.sort.flink.Entrance
inlong-sort/sort-[version].jar \
--cluster-id debezium2hive --dataflow.info.file
/YOUR_DATAFLOW_INFO_DIR/debezium-to-hive.json \
--source.type pulsar --sink.type hive
--sink.hive.rolling-policy.rollover-interval 60000 \
---sink.hive.rolling-policy.check-interval 30000
+--metrics.audit.proxy.hosts 127.0.0.1:10081
--sink.hive.rolling-policy.check-interval 30000
```
Notice：
@@ -36,6 +36,7 @@ Notice：
- `--dataflow.info.file` dataflow configuration file path
- `--source.type` source of the application, currently "pulsar" is supported
- `--sink.type` sink of the application, currently "clickhouse", "hive",
"iceberg", "kafka" are supported
+- `--metrics.audit.proxy.hosts` audit proxy host address for reporting audit
metrics
**Example**
```
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/deployment/k8s.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/deployment/k8s.md
index f7c22de6f..fa2df15e2 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/deployment/k8s.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/deployment/k8s.md
@@ -18,7 +18,48 @@ helm upgrade inlong --install -n inlong ./
```
## 配置
+配置内容都在 **values.yaml** 文件中，以下为所有可配置项及其默认值，包括：
+| Parameter
| Default |
Description
|
+|:--------------------------------------------------------------------------------:|:----------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------:|
+| `timezone`
| `Asia/Shanghai` |
World time and date for cities in all time zones
|
+| `images.pullPolicy`
| `IfNotPresent` | Image
pull policy. One of `Always`, `Never`, `IfNotPresent`
|
+| `images.<component>.repository`
| |
Docker image repository for the component
|
+| `images.<component>.tag`
| `latest` |
Docker image tag for the component
|
+| `<component>.component`
| |
Component name
|
+| `<component>.replicaCount`
| `1` |
Replicas is the desired number of replicas of a given Template
|
+| `<component>.podManagementPolicy`
| `OrderedReady` | PodManagementPolicy controls how pods
are created during initial scale up, when replacing pods on nodes, or when
scaling down |
+| `<component>.annotations`
| `{}` | The `annotations`
field can be used to attach arbitrary non-identifying metadata to objects
|
+| `<component>.tolerations`
| `[]` | Tolerations are applied to pods,
and allow (but do not require) the pods to schedule onto nodes with matching
taints |
+| `<component>.nodeSelector`
| `{}` | You can add the `nodeSelector` field
to your Pod specification and specify the node labels you want the target node
to have |
+| `<component>.affinity`
| `{}` | Node affinity is conceptually similar to
nodeSelector, allowing you to constrain which nodes your Pod can be scheduled
on based on node labels |
+| `<component>.terminationGracePeriodSeconds`
| `30` | Optional
duration in seconds the pod needs to terminate gracefully
|
+| `<component>.resources`
| `{}` |
Optionally specify how much of each resource a container needs
|
+| `<component>.port(s)`
| |
The port(s) for each component service
|
+| `<component>.env`
| `{}` |
Environment variables for each component container
|
+| <code>\<component\>.probe.\<liveness|readiness\>.enabled</code>
| `true` |
Turn on and off liveness or readiness probe
|
+|
<code>\<component\>.probe.\<liveness|readiness\>.failureThreshold</code>
| `10` |
Minimum consecutive successes for the probe
|
+|
<code>\<component\>.probe.\<liveness|readiness\>.initialDelaySeconds</code>
| `10` |
Delay before the probe is initiated
|
+|
<code>\<component\>.probe.\<liveness|readiness\>.periodSeconds</code> |
`30` |
How often to perform the probe
|
+| `<component>.volumes.name`
| |
Volume name
|
+| `<component>.volumes.size`
| `10Gi` |
Volume size
|
+| `<component>.service.annotations`
| `{}` | The
`annotations` field may need to be set when service.type is `LoadBalancer`
|
+| `<component>.service.type`
| `ClusterIP` | The `type` field determines how the
service is exposed. Valid options are `ClusterIP`, `NodePort`, `LoadBalancer`
and `ExternalName` |
+| `<component>.service.clusterIP`
| `nil` | ClusterIP is the IP
address of the service and is usually assigned randomly by the master
|
+| `<component>.service.nodePort`
| `nil` | NodePort is the port on
each node on which this service is exposed when service type is `NodePort`
|
+| `<component>.service.loadBalancerIP`
| `nil` | LoadBalancer will get
created with the IP specified in this field when service type is `LoadBalancer`
|
+| `<component>.service.externalName`
| `nil` | ExternalName is the external reference that kubedns or
equivalent will return as a CNAME record for this service, requires service
type to be `ExternalName` |
+| `<component>.service.externalIPs`
| `[]` | ExternalIPs is a list of IP
addresses for which nodes in the cluster will also accept traffic for this
service |
+| `external.mysql.enabled`
| `false` | If not exists
external MySQL, InLong will use the internal MySQL by default
|
+| `external.mysql.hostname`
| `localhost` |
External MySQL hostname
|
+| `external.mysql.port`
| `3306` |
External MySQL port
|
+| `external.mysql.username`
| `root` |
External MySQL username
|
+| `external.mysql.password`
| `password` |
External MySQL password
|
+| `external.pulsar.enabled`
| `false` | If not exists
external Pulsar, InLong will use the internal TubeMQ by default
|
+| `external.pulsar.serviceUrl`
| `localhost:6650` |
External Pulsar service URL
|
+| `external.pulsar.adminUrl`
| `localhost:8080` |
External Pulsar admin URL
|
+
+> The components include `agent`, `audit`, `dashboard`, `dataproxy`,
`manager`, `tubemq-manager`, `tubemq-master`, `tubemq-broker`, `zookeeper` and
`mysql`.
## 卸载
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/manager/quick_start.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/manager/quick_start.md
index b14d50c0f..cf2362f42 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/manager/quick_start.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/manager/quick_start.md
@@ -17,7 +17,6 @@ title: 安装部署
- 如果后端连接 PostgreSQL 数据库，不需要引入额外依赖。
## 配置
-
前往 `inlong-manager` 目录，修改 `conf/application.properties` 文件：
```properties
@@ -36,6 +35,17 @@ spring.datasource.druid.username=root
spring.datasource.druid.password=inlong
```
+## Flink 插件
+InLong 支持 Manager 发起 Sort 任务进行数据分拣，需要先配置 Flink
环境信息。配置文件为`plugins/flink-sort-plugin.properties`.
+```properties
+# Flink host split by coma if more than one host, such as 'host1,host2'
+flink.rest.address=127.0.0.1
+# Flink port
+flink.rest.port=8081
+# Flink jobmanager port
+flink.jobmanager.port=6123
+```
+
## 启动
```shell
bash +x bin/startup.sh
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/sort/quick_start.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/sort/quick_start.md
index 74ccf066e..0e501049c 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/sort/quick_start.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/sort/quick_start.md
@@ -21,7 +21,7 @@ flink环境配置完成后，可以通过浏览器访问flink的web ui，对应
./bin/flink run -c org.apache.inlong.sort.flink.Entrance
inlong-sort/sort-[version].jar \
--cluster-id debezium2hive --dataflow.info.file
/YOUR_DATAFLOW_INFO_DIR/debezium-to-hive.json \
--source.type pulsar --sink.type hive
--sink.hive.rolling-policy.rollover-interval 60000 \
---sink.hive.rolling-policy.check-interval 30000
+--metrics.audit.proxy.hosts 127.0.0.1:10081
--sink.hive.rolling-policy.check-interval 30000
```
注意：
@@ -35,6 +35,7 @@ flink环境配置完成后，可以通过浏览器访问flink的web ui，对应
- `--dataflow.info.file` 流配置文件路径
- `--source.type` 数据源的种类, 当前支持："pulsar"
- `--sink.type` 存储系统的种类，当前支持："clickhouse"、"hive"、"iceberg"、"kafka"
+- `--metrics.audit.proxy.hosts` audit proxy 地址用于上报审计指标数据
**启动参数配置示例**
```
diff --git a/package.json b/package.json
index cdf71d654..4368f0c35 100644
--- a/package.json
+++ b/package.json
@@ -15,10 +15,10 @@
"write-heading-ids": "docusaurus write-heading-ids"
},
"dependencies": {
- "@docusaurus/core": "^2.0.0-beta.17",
- "@docusaurus/plugin-content-blog": "^2.0.0-beta.17",
- "@docusaurus/plugin-content-docs": "^2.0.0-beta.17",
- "@docusaurus/preset-classic": "2.0.0-beta.17",
+ "@docusaurus/core": "^2.0.0-beta.18",
+ "@docusaurus/plugin-content-blog": "^2.0.0-beta.18",
+ "@docusaurus/plugin-content-docs": "^2.0.0-beta.18",
+ "@docusaurus/preset-classic": "2.0.0-beta.18",
"@mdx-js/react": "^1.6.21",
"@svgr/webpack": "^5.5.0",
"acorn": "^8.6.0",
