This is an automated email from the ASF dual-hosted git repository.
benjobs pushed a commit to branch dev
in repository
https://gitbox.apache.org/repos/asf/incubator-streampark-website.git
The following commit(s) were added to refs/heads/dev by this push:
new 6f9c8fa5 Update the StreamPark configuration doc (#151)
6f9c8fa5 is described below
commit 6f9c8fa57bf8b3d32a8707414d9dc6f4f74e08a9
Author: 1996fanrui <[email protected]>
AuthorDate: Mon Oct 10 01:17:45 2022 +0800
Update the StreamPark configuration doc (#151)
---
docs/components/TableData.jsx | 4 +-
docs/components/data.js | 20 ++++----
docs/development/conf.md | 108 +++++++++++++++++++-----------------------
3 files changed, 60 insertions(+), 72 deletions(-)
diff --git a/docs/components/TableData.jsx b/docs/components/TableData.jsx
index 5ba04513..81366116 100644
--- a/docs/components/TableData.jsx
+++ b/docs/components/TableData.jsx
@@ -31,9 +31,9 @@ const ClientOption = () => {
{
item.deprecated
?
- <span className="icon-check"></span>
- :
<span className="icon-times"></span>
+ :
+ <span className="icon-check"></span>
}
</td>
<td>{item.value}</td>
diff --git a/docs/components/data.js b/docs/components/data.js
index 4964d544..69aaed19 100644
--- a/docs/components/data.js
+++ b/docs/components/data.js
@@ -43,9 +43,7 @@ export default {
{group: 'JM heap Memory', name: 'jobmanager.memory.heap.size', desc:
'JVM Heap Memory size for JobManager. The minimum recommended JVM Heap size is
128.000mb (134217728 bytes).'},
{group: 'JM Off-heap Memory', name: 'jobmanager.memory.off-heap.size',
desc: 'Off-heap Memory size for JobManager. This option covers all off-heap
memory usage including direct and native memory allocation. The JVM direct
memory limit of the JobManager process (-XX:MaxDirectMemorySize) will be set to
this value if the limit is enabled by
jobmanager.memory.enable-jvm-direct-memory-limit'},
{group: 'JVM Metaspace', name: 'jobmanager.memory.jvm-metaspace.size',
desc: 'JVM Metaspace Size for the JobManager.'},
- {group: 'JVM Metaspace', name: 'jobmanager.memory.jvm-metaspace.size',
desc: 'JVM Metaspace Size for the JobManager.'},
- {group: 'JVM Metaspace', name: 'jobmanager.memory.jvm-overhead.min',
desc: 'Min JVM Overhead size for the JobManager. This is off-heap memory
reserved for JVM overhead, such as thread stack space, compile cache, etc. This
includes native memory but not direct memory, and will not be counted when
Flink calculates JVM max direct memory size parameter. The size of JVM Overhead
is derived to make up the configured fraction of the Total Process Memory. If
the derived size is less or g [...]
- {group: 'JVM Size', name: 'jobmanager.memory.jvm-metaspace.size',
desc: 'JVM Metaspace Size for the JobManager.'},
+ {group: 'JVM Size', name: 'jobmanager.memory.jvm-overhead.min', desc:
'Min JVM Overhead size for the JobManager. This is off-heap memory reserved for
JVM overhead, such as thread stack space, compile cache, etc. This includes
native memory but not direct memory, and will not be counted when Flink
calculates JVM max direct memory size parameter. The size of JVM Overhead is
derived to make up the configured fraction of the Total Process Memory. If the
derived size is less or greate [...]
{group: 'JVM Size', name: 'jobmanager.memory.jvm-overhead.max', desc:
'Max JVM Overhead size for the JobManager. This is off-heap memory reserved for
JVM overhead, such as thread stack space, compile cache, etc. This includes
native memory but not direct memory, and will not be counted when Flink
calculates JVM max direct memory size parameter. The size of JVM Overhead is
derived to make up the configured fraction of the Total Process Memory. If the
derived size is less or greate [...]
{group: 'JVM Size', name: 'jobmanager.memory.jvm-overhead.fraction',
desc: 'Fraction of Total Process Memory to be reserved for JVM Overhead. This
is off-heap memory reserved for JVM overhead, such as thread stack space,
compile cache, etc. This includes native memory but not direct memory, and will
not be counted when Flink calculates JVM max direct memory size parameter. The
size of JVM Overhead is derived to make up the configured fraction of the Total
Process Memory. If the d [...]
{group: 'Framework Heap Memory', name:
'taskmanager.memory.framework.heap.size', desc: 'Framework Heap Memory size for
TaskExecutors. This is the size of JVM heap memory reserved for TaskExecutor
framework, which will not be allocated to task slots.'},
@@ -69,17 +67,15 @@ export default {
{group: 'Flink process total memory', tm:
'taskmanager.memory.process.size', jm: 'jobmanager.memory.process.size'}
],
checkpoints: [
- {name: 'enable', desc: 'Whether to open checkpoint', value: 'true |
false'},
- {name: 'interval', desc: 'Interval period of checkpoint', value:
'millisecond'},
- {name: 'mode', desc: 'semantics', value: ' EXACTLY_ONCE |
AT_LEAST_ONCE '},
- {name: 'timeout', desc: 'timeout', value: 'millisecond'},
- {name: 'unaligned', desc: 'unaligned', value: 'true | false'},
+ {name: 'execution.checkpointing.interval', desc: 'Interval period of
checkpoint', value: 'Duration'},
+ {name: 'execution.checkpointing.timeout', desc: 'timeout', value:
'Duration'},
+ {name: 'execution.checkpointing.mode', desc: 'semantics', value: '
EXACTLY_ONCE | AT_LEAST_ONCE '},
+ {name: 'execution.checkpointing.unaligned', desc: 'unaligned', value:
'true | false'}
],
backend: [
- {name: 'value', desc: 'Type of backend storage', value: 'jobmanager |
filesystem | rocksdb', mode: ''},
- {name: 'memory', desc: 'Maximum memory, only valid when the vlue is
jobmanager', value: 'kb(5242880)', mode: 'jobmanager'},
- {name: 'async', desc: 'Whether to enable asynchrony', value: ' true |
false', mode: 'jobmanager | filesystem'},
- {name: 'incremental', desc: 'Whether to enable increment', value: '
true | false', mode: 'rocksdb'},
+ {name: 'state.backend', desc: 'Type of backend storage', value:
'hashmap | rocksdb', mode: ''},
+ {name: 'state.checkpoint-storage', desc: 'The checkpoint storage
implementation to be used to checkpoint state.', value: 'jobmanager |
filesystem ', mode: ''},
+ {name: 'state.backend.incremental', desc: 'Whether to enable
increment', value: ' true | false', mode: 'rocksdb'}
],
fixedDelay: [
{name: 'attempts', desc: 'Number of Flink attempts to restart', value:
'3'},
diff --git a/docs/development/conf.md b/docs/development/conf.md
index 72a3f12c..b45b0a7b 100755
--- a/docs/development/conf.md
+++ b/docs/development/conf.md
@@ -127,7 +127,7 @@ flink:
jobmanager:
property: #@see:
https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/config.html
$internal.application.main:
org.apache.streampark.flink.quickstart.QuickStartApp
- yarn.application.name: Streamx QuickStart App
+ pipeline.name: Streamx QuickStart App
yarn.application.queue:
taskmanager.numberOfTaskSlots: 1
parallelism.default: 2
@@ -150,49 +150,47 @@ flink:
jvm-overhead.max:
jvm-overhead.min:
managed.fraction: 0.4
- checkpoints:
- enable: true
- interval: 30000
- mode: EXACTLY_ONCE
- timeout: 300000
- unaligned: true
- watermark:
- interval: 10000
- # State backend
- state:
- backend: # see
https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/state/state_backends.html
- value: filesystem # 'jobmanager', 'filesystem', 'rocksdb'
- memory: 5242880 # Maximum memory, only valid when the vlue is jobmanager
- async: false # Whether to enable asynchrony is only valid when value
is jobmanager and filesystem
- incremental: true # Whether to enable increment is only valid when value
is rocksdb
- #Configuration reference of rocksdb
https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/config.html#rocksdb-state-backend
- #Rocksdb configures the prefix of the key and removes state.backend
- checkpoints.dir: file:///tmp/chkdir
- savepoints.dir: file:///tmp/chkdir
- # Restart policy
- restart-strategy:
- value: fixed-delay # (fixed-delay|failure-rate|none)
- fixed-delay:
- attempts: 3
- delay: 5000
- failure-rate:
- max-failures-per-interval:
- failure-rate-interval:
- delay:
+ pipeline:
+ auto-watermark-interval: 200ms
+ # checkpoint
+ execution:
+ checkpointing:
+ mode: EXACTLY_ONCE
+ interval: 30s
+ timeout: 10min
+ unaligned: false
+ externalized-checkpoint-retention: RETAIN_ON_CANCELLATION
+ # state backend
+ state:
+ backend: hashmap # Special note: flink1.12 optional configuration
('jobmanager', 'filesystem', 'rocksdb'), flink1.12+ optional configuration
('hashmap', 'rocksdb'),
+ backend.incremental: true
+ checkpoint-storage: filesystem
+ savepoints.dir: file:///tmp/chkdir
+ checkpoints.dir: file:///tmp/chkdir
+ # restart strategy
+ restart-strategy: fixed-delay # Restart strategy
[(fixed-delay|failure-rate|none) a total of 3 configurable strategies]
+ restart-strategy.fixed-delay:
+ attempts: 3
+ delay: 5000
+ restart-strategy.failure-rate:
+ max-failures-per-interval:
+ failure-rate-interval:
+ delay:
+ # table
table:
planner: blink # (blink|old|any)
mode: streaming #(batch|streaming)
```
The above is the complete configuration related to the environment that needs
to be paid attention to. These configurations are carried out under the
namespace of `Flink`, mainly including two categories.
-* The configuration under deployment is the configuration related to the
project `deployment` (`that is, the configuration parameters related to a
series of resources when the project is started`).
+* The configuration under deployment is the configuration related to the
project `deployment` (`that is, the configuration parameters related to a
series of resources when the application is started`).
* Others are the configuration of the environment that needs attention during
development.
-There are five configurations related to the environment that need to be paid
attention to during development.
+There are some configurations related to the environment that need to be paid
attention to during development.
-* `checkpoints`
+* `checkpoint`
* `watermark`
-* `state`
+* `state backend`
* `restart-strategy`
* `table`
@@ -226,10 +224,10 @@ There are many basic parameters. The five most basic
parameters are as follows.
<ClientProperty></ClientProperty>
:::info Attention
-`$internal.application.main` and `yarn.application.name` must be set.
+`$internal.application.main` and `pipeline.name` must be set.
:::
-If you need to set more parameters, please refer to
[`here`](https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/config.html),
These parameters must be placed under the property and the parameter names
must be correct. Streamx will automatically resolve these parameters and take
effect.
+If you need to set more parameters, please refer to
[`here`](https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/config.html),
These parameters must be placed under the property and the parameter names
must be correct. StreamPark will automatically resolve these parameters and
take effect.
##### Memory parameters
@@ -255,30 +253,25 @@ The simplest way to set up memory in Flink is to
configure either of the two fol
Explicitly configuring both total process memory and total Flink memory is not
recommended. It may lead to deployment failures due to potential memory
configuration conflicts. Configuring other memory components also requires
caution as it can produce further configuration conflicts.
:::
-### Checkpoints
+### Checkpoint
-The configuration of checkpoints is simple. You can configure them as follows:
+The configuration of checkpoint is simple. You can configure them as follows:
<ClientCheckpoints></ClientCheckpoints>
### Watermark
-For `watermark` configuration, you only need to set the generation cycle
`interval` of the watermark.
+For `watermark` configuration, you only need to set the generation cycle
`pipeline.auto-watermark-interval` of the watermark.
### State
```yaml
state:
- backend: # see
https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/state/state_backends.html
- value: filesystem # jobmanager, filesystem, rocksdb
- memory: 5242880 # Maximum memory, only valid when the vlue is jobmanager
- async: false # Whether to enable asynchrony is only valid when value is
jobmanager and filesystem
- incremental: true # Whether to enable increment is only valid when value
is rocksdb
- #Configuration reference of rocksdb
https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/config.html#rocksdb-state-backend
- #Rocksdb configures the prefix of the key and removes state.backend
- checkpoints.dir: file:///tmp/chkdir
+ backend: hashmap # Special note: flink1.12 optional configuration
('jobmanager', 'filesystem', 'rocksdb'), flink1.12+ optional configuration
('hashmap', 'rocksdb'),
+ backend.incremental: true
+ checkpoint-storage: filesystem
savepoints.dir: file:///tmp/chkdir
- checkpoints.num-retained: 1
+ checkpoints.dir: file:///tmp/chkdir
```
There are roughly two types:
* backend
@@ -301,18 +294,17 @@ The value item is a non-standard configuration. This item
is used to set the sta
There are three restart strategies in Flink, corresponding to the three
configurations here, as follows:
```yaml
- restart-strategy:
- value: fixed-delay #Restart strategy[(fixed-delay|failure-rate|none)]
- fixed-delay:
- attempts: 3
- delay: 5000
- failure-rate:
- max-failures-per-interval:
- failure-rate-interval:
- delay:
+restart-strategy: fixed-delay # Restart strategy
[(fixed-delay|failure-rate|none) a total of 3 configurable strategies]
+restart-strategy.fixed-delay:
+ attempts: 3
+ delay: 5000
+restart-strategy.failure-rate:
+ max-failures-per-interval:
+ failure-rate-interval:
+ delay:
```
-Configure the specific restart strategy under `value`
+Configure the specific restart strategy under `restart-strategy`
* fixed-delay
* failure-rate
