mfyuce opened a new issue, #3815:
URL: https://github.com/apache/iggy/issues/3815
### Description
Getting OpenTelemetry data into Iggy currently means running a separate
bridge
process in front of it. Collectors and SDKs already speak OTLP/gRPC, and the
connectors runtime already hosts long-lived plugins, so the bridge is
avoidable:
the OTLP service can terminate inside the runtime and produce straight onto
Iggy
topics.
OTLP is listed as a P0 source and sink on the connector roadmap (#2753).
### Affected area / component
Connectors
### Proposed solution
An `otlp_source` connector plugin that binds the OTLP standard port and
serves
`LogsService`, `MetricsService` and `TraceService` from a single listener,
since
standard OTLP deployments expose one endpoint and splitting them would force
operators to configure three addresses that match no existing tooling.
Decode with `opentelemetry-proto`, which already ships the generated tonic
stubs, so no `build.rs` or `protoc` enters the build.
Two storage formats:
- `json`, one queryable document per record, for sinks that need field-level
access.
- `proto`, the raw request bytes as a single message, roughly 4-5x smaller
and
zero-copy when paired with an OTLP sink.
Backpressure must not fail the export. When the in-process channel fills,
enqueue
what fits and report the remainder in the response's `partial_success`,
counted
in OTLP records rather than internal messages, so a client retries only what
was
actually rejected instead of resending a batch that was half delivered.
### Alternatives considered
Running the OpenTelemetry Collector with a custom exporter. That keeps a
second
process in the path and duplicates the connector runtime's lifecycle, config
and
metrics handling.
Generating the protobuf stubs with `tonic-build`. It adds `protoc` to the
build
for types `opentelemetry-proto` already publishes.
### Contribution
Yes, I am willing to submit a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
