maxstreese commented on pull request #3529:
URL: https://github.com/apache/druid/pull/3529#issuecomment-674095377
Hi @gianm,
thanks for that fast reply to my depressive comment. Ok now I also
understand the docs better with the background information that this is
currently only supported in the legacy `parser`. To me personally the docs seem
to not reflect this properly. At least I got quite confused. But in any case I
was able to make it work so let me share my final config for some dummy topic
in case someone else stumbles across this:
```json
{
"type": "kafka",
"dataSchema": {
"dataSource": "ticks",
"parser": {
"type": "avro_stream",
"avroBytesDecoder": {
"type": "schema_registry",
"url": "<schema registry url>"
},
"parseSpec": {
"format": "avro",
"flattenSpec": {
"fields": [
{"name": "instrument", "type": "path", "expr":
"$.id.instrument"},
{"name": "currency", "type": "path", "expr": "$.id.currency"}
]
},
"timestampSpec": {
"column": "timestamp",
"format": "millis"
},
"dimensionsSpec": {
"dimensions": [
"instrument",
"currency",
{"name": "value", "type": "double"}
]
}
}
}
},
"ioConfig": {
"type": "kafka",
"topic": "ticks",
"consumerProperties": {
"bootstrap.servers": "<bootstrap server addresses>"
}
},
"tuningConfig": {
"type": "kafka",
"logParseExceptions": true
}
}
```
The above assumes that there is a topic named `ticks` in your cluster which
contains data encoded with the following Avro schema:
```json
{
"type": "record",
"name": "Tick",
"namespace": "<some namespace>",
"fields": [{
"name": "id",
"type": {
"type": "record",
"name": "Id",
"fields": [{
"name": "instrument",
"type": "string"
}, {
"name": "currency",
"type": "string"
}]
}
}, {
"name": "timestamp",
"type": {
"type": "long",
"logicalType": "timestamp-millis"
}
}, {
"name": "value",
"type": "double"
}]
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
