GitHub user pavelekshin edited a discussion: Airflow 2.10.X and standalone OpenTelemetry Collector integration
I'm trying to integrate Airflow 2.10.5 with a standalone OpenTelemetry Collector version 0.70, following the suggested setup script https://github.com/apache/airflow/blob/2.10.2/scripts/ci/docker-compose/integration-otel.yml. However, I'm encountering a 400 Bad Request error. The OpenTelemetry Collector logs don't show any entries related to the HTTP request. Any ideas on what might be going wrong or how to debug this further? ```logs Airflow side: [Metric Exporter] Connecting to OpenTelemetry Collector at http://otel-collector:4318/v1/metrics [2026-03-17T10:16:03.577+0000] {__init__.py:198} ERROR - Failed to export span batch code: 400, reason: Bad Request [2026-03-17T10:16:03.698+0000] {__init__.py:198} ERROR - Failed to export span batch code: 400, reason: Bad Request [2026-03-17T10:16:08.585+0000] {__init__.py:198} ERROR - Failed to export span batch code: 400, reason: Bad Request [2026-03-17T10:16:08.708+0000] {__init__.py:198} ERROR - Failed to export span batch code: 400, reason: Bad Request Otel side: nothing ``` I also try gRPC via 4317, but airflow not switch to gRPC protocol and continuously send HTTP requests ```logs Airflow side: [2026-03-17T09:06:14.263+0000] {connectionpool.py:241} DEBUG - Starting new HTTP connection (1): otel-collector:4317 [2026-03-17T09:06:14.267+0000] {connectionpool.py:241} DEBUG - Starting new HTTP connection (2): otel-collector:4317 [2026-03-17T09:06:14.267+0000] {__init__.py:266} WARNING - Transient error ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) encountered while exporting metrics batch, retrying in 0.98s Otel side: Closing: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"POST /v1/metrics HTTP/1.\"" ``` This works and return 200, but nothing on otel logs ```logs curl -v -X POST http://otel-collector:4318/v1/metrics -H "Content-Type: application/json" -d '{"resourceMetrics": [{"resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "test"}}]}}]}' ``` I also tested with the latest OpenTelemetry Collector (v0.146). The error changed from 400 to 500 Internal Server Error, but the collector logs remain completely empty. No entries appear even with debug logging enabled. I tried tune exporter settings on airflow side using protocol settings - https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/ OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf, etc but without any success I use following airflow ENV: ```env - AIRFLOW__METRICS__OTEL_ON=True - AIRFLOW__METRICS__OTEL_HOST=otel-collector - AIRFLOW__METRICS__OTEL_PORT=4318 - AIRFLOW__METRICS__OTEL_INTERVAL_MILLISECONDS=30000 - AIRFLOW__TRACES__OTEL_ON=True - AIRFLOW__TRACES__OTEL_HOST=otel-collector - AIRFLOW__TRACES__OTEL_PORT=4318 - AIRFLOW__TRACES__OTEL_DEBUGGING_ON=False - AIRFLOW__TRACES__OTEL_TASK_LOG_EVENT=True ``` and installed otel plugin via apache-airflow[otel] ```bash opentelemetry-api 1.40.0 opentelemetry-exporter-otlp 1.40.0 opentelemetry-exporter-otlp-proto-common 1.40.0 opentelemetry-exporter-otlp-proto-grpc 1.40.0 opentelemetry-exporter-otlp-proto-http 1.40.0 opentelemetry-exporter-prometheus 0.61b0 opentelemetry-proto 1.40.0 opentelemetry-sdk 1.40.0 opentelemetry-semantic-conventions 0.61b0 ``` otel/prometheus setup ```yml otel-collector: image: otel/opentelemetry-collector-contrib:0.70.0 command: [--config=/etc/otel-collector-config.yml] volumes: - ./otel-collector-config.yml:/etc/otel-collector-config.yml ports: - "24318:4318" - "28889:8889" prometheus: image: prom/prometheus user: "0" ports: - "29090:9090" volumes: - ./prometheus-config.yml:/etc/prometheus/prometheus.yml ``` otel config: ```yml receivers: otlp: protocols: http: endpoint: 0.0.0.0:4318 grpc: endpoint: 0.0.0.0:4317 processors: batch: exporters: logging: verbosity: detailed prometheus: endpoint: 0.0.0.0:8889 service: telemetry: logs: level: debug development: true pipelines: metrics: receivers: [otlp] processors: [batch] exporters: [logging, prometheus] traces: receivers: [otlp] processors: [batch] exporters: [logging] ``` GitHub link: https://github.com/apache/airflow/discussions/63808 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
