sashidhar commented on a change in pull request #8098: Kafka lookup custom Json
and Jq extractor
URL: https://github.com/apache/incubator-druid/pull/8098#discussion_r305017262
##########
File path:
docs/content/development/extensions-core/kafka-extraction-namespace.md
##########
@@ -36,23 +36,74 @@ If you need updates to populate as promptly as possible,
it is possible to plug
{
"type":"kafka",
"kafkaTopic":"testTopic",
- "kafkaProperties":{"zookeeper.connect":"somehost:2181/kafka"}
+ "kafkaProperties":{"bootstrap.servers":"brokerhost:9091"},
+ "namespaceParseSpec": {
+ "format": "customJson",
+ "keyFieldName": "id",
+ "valueFieldName": "email"
+ }
}
```
|Parameter|Description|Required|Default|
|---------|-----------|--------|-------|
|`kafkaTopic`|The kafka topic to read the data from|Yes||
-|`kafkaProperties`|Kafka consumer properties. At least"zookeeper.connect" must
be specified. Only the zookeeper connector is supported|Yes||
+|`kafkaProperties`|Kafka consumer properties. At least"bootstrap.servers" must
be specified. |Yes||
|`connectTimeout`|How long to wait for an initial connection|No|`0` (do not
wait)|
|`isOneToOne`|The map is a one-to-one (see [Lookup
DimensionSpecs](../../querying/dimensionspecs.html))|No|`false`|
+|`namespaceParseSpec`|How to extract key and value pairs from Kafka
message|No|null|
The extension `kafka-extraction-namespace` enables reading from a kafka feed
which has name/key pairs to allow renaming of dimension values. An example use
case would be to rename an ID to a human readable format.
-The consumer properties `group.id` and `auto.offset.reset` CANNOT be set in
`kafkaProperties` as they are set by the extension as
`UUID.randomUUID().toString()` and `smallest` respectively.
+The consumer properties `group.id` and `auto.offset.reset` CANNOT be set in
`kafkaProperties` as they are set by the extension as
`UUID.randomUUID().toString()` and `earliest` respectively.
See [lookups](../../querying/lookups.html) for how to configure and use
lookups.
+# Json Extrators
+
+Besides simple kafka key and message mapping, extraction allows to extract
values from Kafka JSON messages.
+
+## Custom Json Extractor
+Allows to extract key and value from Kafka json message as per the following
example:
+
+```json
+{
+ "type": "kafka",
+ "kafkaTopic": "topic_employee",
+ "kafkaProperties": {
+ "bootstrap.servers": "broker:9092"
+ },
+ "namespaceParseSpec": {
+ "format": "customJson",
+ "keyFieldName": "id",
+ "valueFieldName": "email"
+ }
+}
+```
+
+## Jq Json Extractor
+
+Allows to extract key and value from Kafka json message using [JQ
expressions](https://github.com/stedolan/jq/wiki/Cookbook). In the following
example, Json array of categories parsed into a string where each categorie id
get maps to the coresponding categorie name.
Review comment:
Typo categorie.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
