This is an automated email from the ASF dual-hosted git repository.
ic4y pushed a commit to branch main
in repository
https://gitbox.apache.org/repos/asf/incubator-seatunnel-website.git
The following commit(s) were added to refs/heads/main by this push:
new e421a791c2 [Doc]add st engine to start v2 (#164)
e421a791c2 is described below
commit e421a791c262eebb1b1813aa77b0b40f50aa1498
Author: Eric <[email protected]>
AuthorDate: Tue Nov 1 10:55:04 2022 +0800
[Doc]add st engine to start v2 (#164)
* add st engine to start v2
* fix links problems
* fix format error
---
.../version-2.2.0-beta/start-v2/local.mdx | 2 +-
.../version-2.3.0-beta/start-v2/docker.md | 8 +
.../version-2.3.0-beta/start-v2/kubernetes.mdx | 270 +++++++++++++++++++++
.../start-v2/local.mdx | 84 +++----
versioned_docs/version-2.3.0-beta/start/local.mdx | 6 +-
5 files changed, 309 insertions(+), 61 deletions(-)
diff --git a/versioned_docs/version-2.2.0-beta/start-v2/local.mdx
b/versioned_docs/version-2.2.0-beta/start-v2/local.mdx
index 26745d6794..301c8d6b94 100644
--- a/versioned_docs/version-2.2.0-beta/start-v2/local.mdx
+++ b/versioned_docs/version-2.2.0-beta/start-v2/local.mdx
@@ -34,7 +34,7 @@ tar -xzvf "apache-seatunnel-incubating-${version}-bin.tar.gz"
<!-- TODO: We should add example module as quick start which is no need for
install Spark or Flink -->
## Step 3: Install connectors plugin
-Since 2.2.0-beta, the binary package does not provide connector dependencies
by default, so when using it for the first time, we need to execute the
following command to install the connector: (Of course, you can also manually
download the connector from [Apache Maven Repository](https://repo.
maven.apache.org/maven2/org/apache/seatunnel/ to download, then manually move
to the seatunnel subdirectory under the connectors directory).
+Since 2.2.0-beta, the binary package does not provide connector dependencies
by default, so when using it for the first time, we need to execute the
following command to install the connector: (Of course, you can also manually
download the connector from [Apache Maven
Repository](https://repo.maven.apache.org/maven2/org/apache/seatunnel/ to
download, then manually move to the seatunnel subdirectory under the connectors
directory).
```bash
sh bin/install_plugin.sh 2.2.0-beta
```
diff --git a/versioned_docs/version-2.3.0-beta/start-v2/docker.md
b/versioned_docs/version-2.3.0-beta/start-v2/docker.md
new file mode 100644
index 0000000000..2553b99779
--- /dev/null
+++ b/versioned_docs/version-2.3.0-beta/start-v2/docker.md
@@ -0,0 +1,8 @@
+---
+sidebar_position: 3
+---
+
+# Set Up with Docker
+
+<!-- TODO -->
+WIP
\ No newline at end of file
diff --git a/versioned_docs/version-2.3.0-beta/start-v2/kubernetes.mdx
b/versioned_docs/version-2.3.0-beta/start-v2/kubernetes.mdx
new file mode 100644
index 0000000000..88e008a7d2
--- /dev/null
+++ b/versioned_docs/version-2.3.0-beta/start-v2/kubernetes.mdx
@@ -0,0 +1,270 @@
+---
+sidebar_position: 4
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Set Up with Kubernetes
+
+This section provides a quick guide to using SeaTunnel with Kubernetes.
+
+## Prerequisites
+
+We assume that you have a local installations of the following:
+
+- [docker](https://docs.docker.com/)
+- [kubernetes](https://kubernetes.io/)
+- [helm](https://helm.sh/docs/intro/quickstart/)
+
+So that the `kubectl` and `helm` commands are available on your local system.
+
+For kubernetes [minikube](https://minikube.sigs.k8s.io/docs/start/) is our
choice, at the time of writing this we are using version v1.23.3. You can start
a cluster with the following command:
+
+```bash
+minikube start --kubernetes-version=v1.23.3
+```
+
+## Installation
+
+### SeaTunnel docker image
+
+To run the image with SeaTunnel, first create a `Dockerfile`:
+
+<Tabs
+ groupId="engine-type"
+ defaultValue="flink"
+ values={[
+ {label: 'Flink', value: 'flink'},
+ ]}>
+<TabItem value="flink">
+
+```Dockerfile
+FROM flink:1.13
+
+ENV SEATUNNEL_VERSION="2.3.0-beta"
+ENV SEATUNNEL_HOME = "/opt/seatunnel"
+
+RUN mkdir -p $SEATUNNEL_HOME
+
+RUN wget
https://archive.apache.org/dist/incubator/seatunnel/${SEATUNNEL_VERSION}/apache-seatunnel-incubating-${SEATUNNEL_VERSION}-bin.tar.gz
+RUN tar -xzvf apache-seatunnel-incubating-${SEATUNNEL_VERSION}-bin.tar.gz
+
+RUN cp -r apache-seatunnel-incubating-${SEATUNNEL_VERSION}/* $SEATUNNEL_HOME/
+RUN rm -rf apache-seatunnel-incubating-${SEATUNNEL_VERSION}*
+RUN rm -rf $SEATUNNEL_HOME/connectors/seatunnel
+```
+
+Then run the following commands to build the image:
+```bash
+docker build -t seatunnel:2.3.0-beta-flink-1.13 -f Dockerfile .
+```
+Image `seatunnel:2.3.0-beta-flink-1.13` need to be present in the host
(minikube) so that the deployment can take place.
+
+Load image to minikube via:
+```bash
+minikube image load seatunnel:2.3.0-beta-flink-1.13
+```
+
+</TabItem>
+</Tabs>
+
+### Deploying the operator
+
+<Tabs
+ groupId="engine-type"
+ defaultValue="flink"
+ values={[
+ {label: 'Flink', value: 'flink'},
+ ]}>
+<TabItem value="flink">
+
+The steps below provide a quick walk-through on setting up the Flink
Kubernetes Operator.
+
+Install the certificate manager on your Kubernetes cluster to enable adding
the webhook component (only needed once per Kubernetes cluster):
+
+```bash
+kubectl create -f
https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.yaml
+```
+Now you can deploy the latest stable Flink Kubernetes Operator version using
the included Helm chart:
+
+```bash
+
+helm repo add flink-operator-repo
https://downloads.apache.org/flink/flink-kubernetes-operator-0.1.0/
+
+helm install flink-kubernetes-operator
flink-operator-repo/flink-kubernetes-operator
+```
+
+You may verify your installation via `kubectl`:
+
+```bash
+kubectl get pods
+NAME READY STATUS
RESTARTS AGE
+flink-kubernetes-operator-5f466b8549-mgchb 1/1 Running 3
(23h ago) 16d
+
+```
+
+</TabItem>
+</Tabs>
+
+## Run SeaTunnel Application
+
+**Run Application:**: SeaTunnel already providers out-of-the-box
[configurations](https://github.com/apache/incubator-seatunnel/tree/dev/config).
+
+<Tabs
+ groupId="engine-type"
+ defaultValue="flink"
+ values={[
+ {label: 'Flink', value: 'flink'},
+ ]}>
+<TabItem value="flink">
+
+In this guide we are going to use
[flink.streaming.conf](https://github.com/apache/incubator-seatunnel/blob/dev/config/flink.streaming.conf.template):
+
+ ```conf
+env {
+ execution.parallelism = 1
+}
+
+source {
+ FakeSourceStream {
+ result_table_name = "fake"
+ field_name = "name,age"
+ }
+}
+
+transform {
+ sql {
+ sql = "select name,age from fake"
+ }
+}
+
+sink {
+ ConsoleSink {}
+}
+ ```
+
+This configuration need to be present when we are going to deploy the
application (SeaTunnel) to Flink cluster (on Kubernetes), we also need to
configure a Pod to Use a PersistentVolume for Storage.
+- Create `/mnt/data` on your Node. Open a shell to the single Node in your
cluster. How you open a shell depends on how you set up your cluster. For
example, in our case weare using Minikube, you can open a shell to your Node by
entering `minikube ssh`.
+In your shell on that Node, create a /mnt/data directory:
+```bash
+minikube ssh
+
+# This assumes that your Node uses "sudo" to run commands
+# as the superuser
+sudo mkdir /mnt/data
+```
+- Copy application (SeaTunnel) configuration files to your Node.
+```bash
+minikube cp flink.streaming.conf /mnt/data/flink.streaming.conf
+```
+
+Once the Flink Kubernetes Operator is running as seen in the previous steps
you are ready to submit a Flink (SeaTunnel) job:
+- Create `seatunnel-flink.yaml` FlinkDeployment manifest:
+```yaml
+apiVersion: flink.apache.org/v1alpha1
+kind: FlinkDeployment
+metadata:
+ namespace: default
+ name: seatunnel-flink-streaming-example
+spec:
+ image: seatunnel:2.3.0-beta-flink-1.13
+ flinkVersion: v1_14
+ flinkConfiguration:
+ taskmanager.numberOfTaskSlots: "2"
+ serviceAccount: flink
+ jobManager:
+ replicas: 1
+ resource:
+ memory: "2048m"
+ cpu: 1
+ taskManager:
+ resource:
+ memory: "2048m"
+ cpu: 2
+ podTemplate:
+ spec:
+ containers:
+ - name: flink-main-container
+ volumeMounts:
+ - mountPath: /data
+ name: config-volume
+ volumes:
+ - name: config-volume
+ hostPath:
+ path: "/mnt/data"
+ type: Directory
+
+ job:
+ jarURI: local:///opt/seatunnel/lib/seatunnel-flink-starter.jar
+ entryClass: org.apache.seatunnel.core.starter.flink.SeatunnelFlink
+ args: ["--config", "/data/flink.streaming.conf"]
+ parallelism: 2
+ upgradeMode: stateless
+
+```
+- Run the example application:
+```bash
+kubectl apply -f seatunnel-flink.yaml
+```
+</TabItem>
+</Tabs>
+
+**See The Output**
+
+<Tabs
+ groupId="engine-type"
+ defaultValue="flink"
+ values={[
+ {label: 'Flink', value: 'flink'},
+ ]}>
+<TabItem value="flink">
+
+You may follow the logs of your job, after a successful startup (which can
take on the order of a minute in a fresh environment, seconds afterwards) you
can:
+
+```bash
+kubectl logs -f deploy/seatunnel-flink-streaming-example
+```
+
+To expose the Flink Dashboard you may add a port-forward rule:
+```bash
+kubectl port-forward svc/seatunnel-flink-streaming-example-rest 8081
+```
+Now the Flink Dashboard is accessible at
[localhost:8081](http://localhost:8081).
+
+Or launch `minikube dashboard` for a web-based Kubernetes user interface.
+
+The content printed in the TaskManager Stdout log:
+```bash
+kubectl logs \
+-l 'app in (seatunnel-flink-streaming-example), component in (taskmanager)' \
+--tail=-1 \
+-f
+```
+looks like the below (your content may be different since we use
`FakeSourceStream` to automatically generate random stream data):
+
+```shell
++I[Kid Xiong, 1650316786086]
++I[Ricky Huo, 1650316787089]
++I[Ricky Huo, 1650316788089]
++I[Ricky Huo, 1650316789090]
++I[Kid Xiong, 1650316790090]
++I[Kid Xiong, 1650316791091]
++I[Kid Xiong, 1650316792092]
+```
+
+To stop your job and delete your FlinkDeployment you can simply:
+
+```bash
+kubectl delete -f seatunnel-flink.yaml
+```
+</TabItem>
+</Tabs>
+
+
+Happy SeaTunneling!
+
+## What's More
+
+For now, you are already taking a quick look at SeaTunnel, you could see
[connector](/category/connector) to find all source and sink SeaTunnel
supported.
+Or see [deployment](../deployment.mdx) if you want to submit your application
in another kind of your engine cluster.
diff --git a/versioned_docs/version-2.2.0-beta/start-v2/local.mdx
b/versioned_docs/version-2.3.0-beta/start-v2/local.mdx
similarity index 77%
copy from versioned_docs/version-2.2.0-beta/start-v2/local.mdx
copy to versioned_docs/version-2.3.0-beta/start-v2/local.mdx
index 26745d6794..1048e04553 100644
--- a/versioned_docs/version-2.2.0-beta/start-v2/local.mdx
+++ b/versioned_docs/version-2.3.0-beta/start-v2/local.mdx
@@ -27,20 +27,20 @@ package `seatunnel-<version>-bin.tar.gz`
Or you can download it by terminal
```shell
-export version="2.2.0-beta"
+export version="2.3.0-beta"
wget
"https://archive.apache.org/dist/incubator/seatunnel/${version}/apache-seatunnel-incubating-${version}-bin.tar.gz";
tar -xzvf "apache-seatunnel-incubating-${version}-bin.tar.gz"
```
<!-- TODO: We should add example module as quick start which is no need for
install Spark or Flink -->
## Step 3: Install connectors plugin
-Since 2.2.0-beta, the binary package does not provide connector dependencies
by default, so when using it for the first time, we need to execute the
following command to install the connector: (Of course, you can also manually
download the connector from [Apache Maven Repository](https://repo.
maven.apache.org/maven2/org/apache/seatunnel/ to download, then manually move
to the seatunnel subdirectory under the connectors directory).
+Since 2.3.0-beta, the binary package does not provide connector dependencies
by default, so when using it for the first time, we need to execute the
following command to install the connector: (Of course, you can also manually
download the connector from [Apache Maven
Repository](https://repo.maven.apache.org/maven2/org/apache/seatunnel/ to
download, then manually move to the seatunnel subdirectory under the connectors
directory).
```bash
-sh bin/install_plugin.sh 2.2.0-beta
+sh bin/install_plugin.sh 2.3.0-beta
```
-If you need to specify the version of the connector, take 2.2.0-beta as an
example, we need to execute
+If you need to specify the version of the connector, take 2.3.0-beta as an
example, we need to execute
```bash
-sh bin/install_plugin.sh 2.2.0-beta
+sh bin/install_plugin.sh 2.3.0-beta
```
Usually we don't need all the connector plugins, so you can specify the
plugins you need by configuring `config/plugin_config`, for example, you only
need the `connector-console` plugin, then you can modify plugin.properties as
```plugin_config
@@ -63,7 +63,7 @@ You can find all supported connectors and corresponding
plugin_config configurat
If you want to install the connector plugin by manually downloading the
connector, you need to pay special attention to the following
-:::
+:::tip
The connectors directory contains the following subdirectories, if they do not
exist, you need to create them manually
@@ -79,27 +79,24 @@ If you want to install the V2 connector plugin manually,
you only need to downlo
## Step 4: Configure SeaTunnel Application
+### Spark or Flink
+
**Configure SeaTunnel**: Change the setting in `config/seatunnel-env.sh`, it
is base on the path your engine install at [prepare step two](#prepare).
Change `SPARK_HOME` if you using Spark as your engine, or change `FLINK_HOME`
if you're using Flink.
+### SeaTunnel Engine
+
+SeaTunnel Engine is the default engine for SeaTunnel, You do not need to do
other additional configuration operations.
+
+### Add Job Config File to define a job
+
Edit `config/seatunnel.streaming.conf.template`, which determines the way and
logic of data input, processing, and output after seatunnel is started.
The following is an example of the configuration file, which is the same as
the example application mentioned above.
```hocon
env {
- # You can set flink configuration here
execution.parallelism = 1
- job.mode = "STREAMING"
- #execution.checkpoint.interval = 10000
- #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
-
-
- # For Spark
- #spark.app.name = "SeaTunnel"
- #spark.executor.instances = 2
- #spark.executor.cores = 1
- #spark.executor.memory = "1g"
- #spark.master = local
+ job.mode = "BATCH"
}
source {
@@ -116,9 +113,7 @@ source {
}
transform {
- sql {
- sql = "select name,age from fake"
- }
+
}
sink {
@@ -139,6 +134,7 @@ You could start the application by the following commands
values={[
{label: 'Spark', value: 'spark'},
{label: 'Flink', value: 'flink'},
+ {label: 'SeaTunnel Engine', value: 'SeaTunnel Engine'},
]}>
<TabItem value="spark">
@@ -159,6 +155,16 @@ cd "apache-seatunnel-incubating-${version}"
--config ./config/seatunnel.streaming.conf.template
```
+</TabItem>
+
+<TabItem value="SeaTunnel Engine">
+
+```shell
+cd "apache-seatunnel-incubating-${version}"
+./bin/seatunnel.sh \
+--config ./config/seatunnel.streaming.conf.template -e local
+```
+
</TabItem>
</Tabs>
@@ -190,42 +196,6 @@ row=16 : SGZCr, 94186144
If use Flink, The content printed in the TaskManager Stdout log of `flink
WebUI`.
-## Explore More Build-in Examples
-
-Our local quick start is using one of the build-in example in directory
`config`, and we provider more than one out-of-box
-example you could and feel free to have a try and make your hands dirty. All
you have to do is change the started command
-option value in [running application](#run-seaTunnel-application) to the
configuration you want to run, we use batch
-template in `config` as examples:
-
-<Tabs
- groupId="engine-type"
- defaultValue="spark"
- values={[
- {label: 'Spark', value: 'spark'},
- {label: 'Flink', value: 'flink'},
- ]}>
-<TabItem value="spark">
-
-```shell
-cd "apache-seatunnel-incubating-${version}"
-./bin/start-seatunnel-spark-connector-v2.sh \
---master local[4] \
---deploy-mode client \
---config ./config/spark.batch.conf.template
-```
-
-</TabItem>
-<TabItem value="flink">
-
-```shell
-cd "apache-seatunnel-incubating-${version}"
-./bin/start-seatunnel-flink-connector-v2.sh \
---config ./config/flink.batch.conf.template
-```
-
-</TabItem>
-</Tabs>
-
## What's More
For now, you are already take a quick look about SeaTunnel, you could see
[connector](/category/connector) to find all
diff --git a/versioned_docs/version-2.3.0-beta/start/local.mdx
b/versioned_docs/version-2.3.0-beta/start/local.mdx
index f9115b6bde..cfc9a1d3ff 100644
--- a/versioned_docs/version-2.3.0-beta/start/local.mdx
+++ b/versioned_docs/version-2.3.0-beta/start/local.mdx
@@ -32,13 +32,13 @@ tar -xzvf
"apache-seatunnel-incubating-${version}-bin.tar.gz"
<!-- TODO: We should add example module as quick start which is no need for
install Spark or Flink -->
## Install connectors plugin
-Since 2.2.0-beta, the binary package does not provide connector dependencies
by default, so when using it for the first time, we need to execute the
following command to install the connector: (Of course, you can also manually
download the connector from [Apache Maven Repository](https://repo.
maven.apache.org/maven2/org/apache/seatunnel/ to download, then manually move
to the connectors directory).
+Since 2.3.0-beta, the binary package does not provide connector dependencies
by default, so when using it for the first time, we need to execute the
following command to install the connector: (Of course, you can also manually
download the connector from [Apache Maven
Repository](https://repo.maven.apache.org/maven2/org/apache/seatunnel/ to
download, then manually move to the connectors directory).
```bash
sh bin/install_plugin.sh
```
-If you need to specify the version of the connector, take 2.2.0-beta as an
example, we need to execute
+If you need to specify the version of the connector, take 2.3.0-beta as an
example, we need to execute
```bash
-sh bin/install_plugin.sh 2.2.0-beta
+sh bin/install_plugin.sh 2.3.0-beta
```
Usually we don't need all the connector plugins, so you can specify the
plugins you need by configuring `config/plugin_config`, for example, you only
need the `flink-assert` plugin, then you can modify plugin.properties as
```plugin_config
