clintropolis commented on a change in pull request #10839:
URL: https://github.com/apache/druid/pull/10839#discussion_r586254344
##########
File path: docs/development/extensions-core/protobuf.md
##########
@@ -72,22 +72,47 @@ message Metrics {
}
```
-### Descriptor file
+### When use descriptor file
Review comment:
```suggestion
### When using a descriptor file
```
##########
File path: docs/development/extensions-core/protobuf.md
##########
@@ -56,7 +56,7 @@ Here is a JSON example of the 'metrics' data schema used in
the example.
### Proto file
-The corresponding proto file for our 'metrics' dataset looks like this.
+The corresponding proto file for our 'metrics' dataset looks like this. You
can use Protobuf parser with a proto file or Confluent schema registry.
Review comment:
```suggestion
The corresponding proto file for our 'metrics' dataset looks like this. You
can use Protobuf parser with a proto file or [Confluent Schema
Registry](https://docs.confluent.io/platform/current/schema-registry/index.html).
```
##########
File path: docs/development/extensions-core/protobuf.md
##########
@@ -72,22 +72,47 @@ message Metrics {
}
```
-### Descriptor file
+### When use descriptor file
Next, we use the `protoc` Protobuf compiler to generate the descriptor file
and save it as `metrics.desc`. The descriptor file must be either in the
classpath or reachable by URL. In this example the descriptor file was saved
at `/tmp/metrics.desc`, however this file is also available in the example
files. From your Druid install directory:
```
protoc -o /tmp/metrics.desc ./quickstart/protobuf/metrics.proto
```
+### When use schema registry
+
+At first make sure your schema registry version is later than 5.5. Next, we
post this schema to schema registry.
Review comment:
```suggestion
Make sure your Schema Registry version is later than 5.5. Next, we can post
a schema to add it to the registry:
```
##########
File path: docs/development/extensions-core/protobuf.md
##########
@@ -164,6 +192,39 @@ Important supervisor properties
}
```
+To adopt to old version. You can use old parser style, which also works.
+
+```json
+{
+ "parser": {
+ "type": "protobuf",
+ "descriptor": "file:///tmp/metrics.desc",
+ "protoMessageType": "Metrics"
+ }
+}
+```
+
+### When use schema registry
Review comment:
```suggestion
### When using Schema Registry
```
##########
File path: docs/development/extensions-core/protobuf.md
##########
@@ -72,22 +72,47 @@ message Metrics {
}
```
-### Descriptor file
+### When use descriptor file
Next, we use the `protoc` Protobuf compiler to generate the descriptor file
and save it as `metrics.desc`. The descriptor file must be either in the
classpath or reachable by URL. In this example the descriptor file was saved
at `/tmp/metrics.desc`, however this file is also available in the example
files. From your Druid install directory:
```
protoc -o /tmp/metrics.desc ./quickstart/protobuf/metrics.proto
```
+### When use schema registry
+
+At first make sure your schema registry version is later than 5.5. Next, we
post this schema to schema registry.
+
+```
+POST /subjects/test/versions HTTP/1.1
+Host: schemaregistry.example.com
+Accept: application/vnd.schemaregistry.v1+json,
application/vnd.schemaregistry+json, application/json
+
+{
+ "schemaType": "PROTOBUF",
+ "schema": "syntax = \"proto3\";\nmessage Metrics {\n string unit = 1;\n
string http_method = 2;\n int32 value = 3;\n string timestamp = 4;\n string
http_code = 5;\n string page = 6;\n string metricType = 7;\n string server =
8;\n}\n"
+}
+```
+
+This feature uses Confluent's Protobuf provider which is not included in the
Druid distribution and must be installed separately. You can fetch it and its
dependencies from Maven Central at:
+-
https://packages.confluent.io/maven/io/confluent/kafka-protobuf-provider/6.0.1/kafka-protobuf-provider-6.0.1.jar
+-
https://repo1.maven.org/maven2/org/jetbrains/kotlin/kotlin-stdlib/1.4.0/kotlin-stdlib-1.4.0.jar
+-
https://repo1.maven.org/maven2/com/squareup/wire/wire-schema/3.2.2/wire-schema-3.2.2.jar
+
+Copy or symlink those files to `extensions/protobuf-extensions` under the
distribution root directory.
+
## Create Kafka Supervisor
Below is the complete Supervisor spec JSON to be submitted to the Overlord.
Make sure these keys are properly configured for successful ingestion.
+### When use descriptor file
Review comment:
```suggestion
### When using a descriptor file
```
##########
File path: docs/development/extensions-core/protobuf.md
##########
@@ -72,22 +72,47 @@ message Metrics {
}
```
-### Descriptor file
+### When use descriptor file
Next, we use the `protoc` Protobuf compiler to generate the descriptor file
and save it as `metrics.desc`. The descriptor file must be either in the
classpath or reachable by URL. In this example the descriptor file was saved
at `/tmp/metrics.desc`, however this file is also available in the example
files. From your Druid install directory:
```
protoc -o /tmp/metrics.desc ./quickstart/protobuf/metrics.proto
```
+### When use schema registry
Review comment:
```suggestion
### When using Schema Registry
```
##########
File path: docs/development/extensions-core/protobuf.md
##########
@@ -72,22 +72,47 @@ message Metrics {
}
```
-### Descriptor file
+### When use descriptor file
Next, we use the `protoc` Protobuf compiler to generate the descriptor file
and save it as `metrics.desc`. The descriptor file must be either in the
classpath or reachable by URL. In this example the descriptor file was saved
at `/tmp/metrics.desc`, however this file is also available in the example
files. From your Druid install directory:
```
protoc -o /tmp/metrics.desc ./quickstart/protobuf/metrics.proto
```
+### When use schema registry
+
+At first make sure your schema registry version is later than 5.5. Next, we
post this schema to schema registry.
+
+```
+POST /subjects/test/versions HTTP/1.1
+Host: schemaregistry.example.com
+Accept: application/vnd.schemaregistry.v1+json,
application/vnd.schemaregistry+json, application/json
+
+{
+ "schemaType": "PROTOBUF",
+ "schema": "syntax = \"proto3\";\nmessage Metrics {\n string unit = 1;\n
string http_method = 2;\n int32 value = 3;\n string timestamp = 4;\n string
http_code = 5;\n string page = 6;\n string metricType = 7;\n string server =
8;\n}\n"
+}
+```
+
+This feature uses Confluent's Protobuf provider which is not included in the
Druid distribution and must be installed separately. You can fetch it and its
dependencies from Maven Central at:
Review comment:
```suggestion
This feature uses Confluent's Protobuf provider which is not included in the
Druid distribution and must be installed separately. You can fetch it and its
dependencies from the Confluent repository and Maven Central at:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
