This is an automated email from the ASF dual-hosted git repository.
cwylie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new 104c9a0 Fix broken anchor and heading levels in Kafka/Kinesis
ingestion (#11748)
104c9a0 is described below
commit 104c9a07f0116b781ec509baab603e9023a52af1
Author: Frank Chen <[email protected]>
AuthorDate: Wed Oct 6 10:30:50 2021 +0800
Fix broken anchor and heading levels in Kafka/Kinesis ingestion (#11748)
* Fix broken anchor and heading levels
* Fix CI
---
docs/development/extensions-core/kafka-ingestion.md | 10 +++++-----
docs/development/extensions-core/kinesis-ingestion.md | 10 +++++-----
2 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/docs/development/extensions-core/kafka-ingestion.md
b/docs/development/extensions-core/kafka-ingestion.md
index dcebce0..d30bf6a 100644
--- a/docs/development/extensions-core/kafka-ingestion.md
+++ b/docs/development/extensions-core/kafka-ingestion.md
@@ -147,9 +147,9 @@ Where the file `supervisor-spec.json` contains a Kafka
supervisor spec:
|`lateMessageRejectionStartDateTime`|ISO8601 DateTime|Configure tasks to
reject messages with timestamps earlier than this date time; for example if
this is set to `2016-01-01T11:00Z` and the supervisor creates a task at
*2016-01-01T12:00Z*, Druid drops messages with timestamps earlier than
*2016-01-01T11:00Z*. This can prevent concurrency issues if your data stream
has late messages and you have multiple pipelines that need to operate on the
same segments (e.g. a realtime and a nightly [...]
|`lateMessageRejectionPeriod`|ISO8601 Period|Configure tasks to reject
messages with timestamps earlier than this period before the task was created;
for example if this is set to `PT1H` and the supervisor creates a task at
*2016-01-01T12:00Z*, messages with timestamps earlier than *2016-01-01T11:00Z*
will be dropped. This may help prevent concurrency issues if your data stream
has late messages and you have multiple pipelines that need to operate on the
same segments (e.g. a realtime an [...]
|`earlyMessageRejectionPeriod`|ISO8601 Period|Configure tasks to reject
messages with timestamps later than this period after the task reached its
taskDuration; for example if this is set to `PT1H`, the taskDuration is set to
`PT1H` and the supervisor creates a task at *2016-01-01T12:00Z*, messages with
timestamps later than *2016-01-01T14:00Z* will be dropped. **Note:** Tasks
sometimes run past their task duration, for example, in cases of supervisor
failover. Setting earlyMessageReject [...]
-|`autoScalerConfig`|Object|Defines auto scaling behavior for Kafka ingest
tasks. See [Tasks Autoscaler Properties](#Task Autoscaler Properties).|no
(default == null)|
+|`autoScalerConfig`|Object|Defines auto scaling behavior for Kafka ingest
tasks. See [Tasks Autoscaler Properties](#task-autoscaler-properties).|no
(default == null)|
-### Task Autoscaler Properties
+#### Task Autoscaler Properties
> Note that Task AutoScaler is currently designated as experimental.
@@ -159,9 +159,9 @@ Where the file `supervisor-spec.json` contains a Kafka
supervisor spec:
| `taskCountMax` | Maximum number of ingestion tasks. Set `taskCountMax >=
taskCountMin`. If `taskCountMax > {numKafkaPartitions}`, Druid only scales
reading tasks up to the `{numKafkaPartitions}`. In this case `taskCountMax` is
ignored. | yes |
| `taskCountMin` | Minimum number of ingestion tasks. When you enable
autoscaler, Druid ignores the value of taskCount in `IOConfig` and starts with
the `taskCountMin` number of tasks.| yes |
| `minTriggerScaleActionFrequencyMillis` | Minimum time interval between two
scale actions. | no (default == 600000) |
-| `autoScalerStrategy` | The algorithm of `autoScaler`. Only supports
`lagBased`. See [Lag Based AutoScaler Strategy Related Properties](#Lag Based
AutoScaler Strategy Related Properties) for details.| no (default ==
`lagBased`) |
+| `autoScalerStrategy` | The algorithm of `autoScaler`. Only supports
`lagBased`. See [Lag Based AutoScaler Strategy Related
Properties](#lag-based-autoscaler-strategy-related-properties) for details.| no
(default == `lagBased`) |
-### Lag Based AutoScaler Strategy Related Properties
+##### Lag Based AutoScaler Strategy Related Properties
| Property | Description | Required |
| ------------- | ------------- | ------------- |
| `lagCollectionIntervalMillis` | Period of lag points collection. | no
(default == 30000) |
@@ -222,7 +222,7 @@ The following example demonstrates supervisor spec with
`lagBased` autoScaler en
#### More on consumerProperties
This must contain a property `bootstrap.servers` with a list of Kafka brokers
in the form: `<BROKER_1>:<PORT_1>,<BROKER_2>:<PORT_2>,...`.
-By default, `isolation.level` is set to `read_committed`. It should be set to
`read_uncommitted` if you don't want Druid to consume only committed
transactions or working with older versions of Kafka servers with no
Transactions support.
+By default, `isolation.level` is set to `read_committed`. It should be set to
`read_uncommitted` if you don't want Druid to consume only committed
transactions or working with older versions of Kafka servers with no
transactions support.
There are few cases that require fetching few/all of consumer properties at
runtime e.g. when `bootstrap.servers` is not known upfront or not static, to
enable SSL connections users might have to provide passwords for `keystore`,
`truststore` and `key` secretly.
For such consumer properties, user can implement a
[DynamicConfigProvider](../../operations/dynamic-config-provider.md) to supply
them at runtime, by adding
diff --git a/docs/development/extensions-core/kinesis-ingestion.md
b/docs/development/extensions-core/kinesis-ingestion.md
index 2a4e1af..2dc60ef 100644
--- a/docs/development/extensions-core/kinesis-ingestion.md
+++ b/docs/development/extensions-core/kinesis-ingestion.md
@@ -146,9 +146,9 @@ Where the file `supervisor-spec.json` contains a Kinesis
supervisor spec:
|`awsAssumedRoleArn`|String|The AWS assumed role to use for additional
permissions.|no|
|`awsExternalId`|String|The AWS external id to use for additional
permissions.|no|
|`deaggregate`|Boolean|Whether to use the de-aggregate function of the KCL.
See below for details.|no|
-|`autoScalerConfig`|Object|Defines auto scaling behavior for Kinesis ingest
tasks. See [Tasks Autoscaler Properties](#Task Autoscaler Properties).|no
(default == null)|
+|`autoScalerConfig`|Object|Defines auto scaling behavior for Kinesis ingest
tasks. See [Tasks Autoscaler Properties](#task-autoscaler-properties).|no
(default == null)|
-### Task Autoscaler Properties
+#### Task Autoscaler Properties
> Note that Task AutoScaler is currently designated as experimental.
@@ -158,9 +158,9 @@ Where the file `supervisor-spec.json` contains a Kinesis
supervisor spec:
| `taskCountMax` | Maximum number of Kinesis ingestion tasks. Must be greater
than or equal to `taskCountMin`. If greater than `{numKinesisShards}`, the
maximum number of reading tasks is `{numKinesisShards}` and `taskCountMax` is
ignored. | yes |
| `taskCountMin` | Minimum number of Kinesis ingestion tasks. When you enable
the auto scaler, Druid ignores the value of taskCount in `IOConfig` and
uses`taskCountMin` for the initial number of tasks to launch.| yes |
| `minTriggerScaleActionFrequencyMillis` | Minimum time interval between two
scale actions | no (default == 600000) |
-| `autoScalerStrategy` | The algorithm of `autoScaler`. ONLY `lagBased` is
supported for now. See [Lag Based AutoScaler Strategy Related Properties](#Lag
Based AutoScaler Strategy Related Properties) for details.| no (default ==
`lagBased`) |
+| `autoScalerStrategy` | The algorithm of `autoScaler`. ONLY `lagBased` is
supported for now. See [Lag Based AutoScaler Strategy Related
Properties](#lag-based-autoscaler-strategy-related-properties) for details.| no
(default == `lagBased`) |
-### Lag Based AutoScaler Strategy Related Properties
+##### Lag Based AutoScaler Strategy Related Properties
The Kinesis indexing service reports lag metrics measured in time milliseconds
rather than message count which is used by Kafka.
@@ -303,7 +303,7 @@ The tuningConfig is optional and default parameters will be
used if no tuningCon
| `intermediateHandoffPeriod` | ISO8601 Period | How often the tasks
should hand off segments. Handoff will happen either if `maxRowsPerSegment` or
`maxTotalRows` is hit or every `intermediateHandoffPeriod`, whichever happens
earlier.
[...]
| `logParseExceptions` | Boolean | If true, log an
error message when a parsing exception occurs, containing information about the
row where the error occurred.
[...]
| `maxParseExceptions` | Integer | The maximum number
of parse exceptions that can occur before the task halts ingestion and fails.
Overridden if `reportParseExceptions` is set.
[...]
-| `maxSavedParseExceptions` | Integer | When a parse
exception occurs, Druid can keep track of the most recent parse exceptions.
"maxSavedParseExceptions" limits how many exception instances will be saved.
These saved exceptions will be made available after the task finishes in the
[task completion report](../../ingestion/tasks.md#reports). Overridden if
`reportParseExceptions` is set.
[...]
+| `maxSavedParseExceptions` | Integer | When a parse
exception occurs, Druid can keep track of the most recent parse exceptions.
"maxSavedParseExceptions" limits how many exception instances will be saved.
These saved exceptions will be made available after the task finishes in the
[task completion report](../../ingestion/tasks.md#task-reports). Overridden if
`reportParseExceptions` is set.
[...]
| `maxRecordsPerPoll` | Integer | The maximum number
of records/events to be fetched from buffer per poll. The actual maximum will
be `Max(maxRecordsPerPoll, Max(bufferSize, 1))`
[...]
| `repartitionTransitionDuration` | ISO8601 Period | When shards are
split or merged, the supervisor will recompute shard -> task group mappings,
and signal any running tasks created under the old mappings to stop early at
(current time + `repartitionTransitionDuration`). Stopping the tasks early
allows Druid to begin reading from the new shards more quickly. The repartition
transition wait time controlled by this property gives the stream additional
time to write records to the ne [...]
| `offsetFetchPeriod` | ISO8601 Period | How often the
supervisor queries Kinesis and the indexing tasks to fetch current offsets and
calculate lag. If the user-specified value is below the minimum value (`PT5S`),
the supervisor ignores the value and uses the minimum value instead.
[...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
