This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
The following commit(s) were added to refs/heads/master by this push:
new fdad954ce IMPALA-13237: [Patch 4 of 5] - Helpers to Visualize
OpenTelemetry Traces
fdad954ce is described below
commit fdad954ce47eaf9944dc742e0c24b0f56262ecc1
Author: jasonmfehr <jf...@cloudera.com>
AuthorDate: Fri Jun 27 12:27:11 2025 -0700
IMPALA-13237: [Patch 4 of 5] - Helpers to Visualize OpenTelemetry Traces
Adds helper scripts and configurations to run an OpenTelemetry OTLP
collector and a Jaeger instance. The collector is configured to
receive telemetry data on port 55888 via OTLP-over-http and to
forward traces to a Jaeger-all-in-one container receiving data on
port 4317.
Testing was accomplished by running this setup locally and verifying traces
appeared in
the Jaeger UI.
Generated-by: Github Copilot (GPT-4.1)
Change-Id: I198c00ddc99a87c630a6f654042bffece2c9d0fd
Reviewed-on: http://gerrit.cloudera.org:8080/23100
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
---
bin/rat_exclude_files.txt | 1 +
testdata/bin/otel-collector/README.md | 110 +++++++++++++++++++++++++
testdata/bin/otel-collector/docker-compose.yml | 43 ++++++++++
testdata/bin/otel-collector/otel-config.yml | 36 ++++++++
4 files changed, 190 insertions(+)
diff --git a/bin/rat_exclude_files.txt b/bin/rat_exclude_files.txt
index da2120786..b2d881961 100644
--- a/bin/rat_exclude_files.txt
+++ b/bin/rat_exclude_files.txt
@@ -104,6 +104,7 @@ README*.md
*/README.txt
testdata/bin/README-BENCHMARK-TEST-GENERATION
testdata/bin/minicluster_lakekeeper/README.md
+testdata/bin/otel-collector/README.md
testdata/scale_test_metadata/README.md
tests/comparison/ORACLE.txt
bin/distcc/README.md
diff --git a/testdata/bin/otel-collector/README.md
b/testdata/bin/otel-collector/README.md
new file mode 100644
index 000000000..b52c818c8
--- /dev/null
+++ b/testdata/bin/otel-collector/README.md
@@ -0,0 +1,110 @@
+# OpenTelemetry Collector & Jaeger Integration
+
+This directory contains configuration to run an [OpenTelemetry
Collector](https://opentelemetry.io/docs/collector/) and a
[Jaeger](https://www.jaegertracing.io/) instance for collecting and visualizing
telemetry, primarily for Impala development and testing.
+
+---
+
+## Contents
+
+- [`otel-config.yml`](./otel-config.yml): OpenTelemetry Collector
configuration file.
+- [`docker-compose.yml`](./docker-compose.yml): Alternative Docker Compose
setup for both services.
+
+---
+
+## Quick Start
+
+### Option 1: Run Interactively
+
+ ```bash
+ docker-compose -f testdata/bin/otel-collector/docker-compose.yml up
+ ```
+
+ - This command:
+ - Starts the OpenTelemetry Collector container with the `otel-config.yml`
config.
+ - Starts the Jaeger container.
+ - Waits for both containers to be running.
+ - Continuously outputs both container logs to the terminal window.
+
+### Option 2: Run Detached
+
+ ```bash
+ docker-compose -f testdata/bin/otel-collector/docker-compose.yml up -d
+ ```
+
+ - This command:
+ - Starts the OpenTelemetry Collector container with the `otel-config.yml`
config.
+ - Starts the Jaeger container.
+ - Waits for both containers to be running.
+ - Exits back to the terminal prompt.
+
+### Stop the Collector and Jaeger
+
+ ```bash
+ docker-compose -f testdata/bin/otel-collector/docker-compose.yml down
+ ```
+
+ - This command gracefully stops and removes the containers and network.
Note this command must always be run. Pressing ctrl+c when the containers are
run interactively does not stop the containers.
+
+---
+
+## Configuration Details
+
+- **OpenTelemetry Collector** listens for OTLP traces on port `55888` (HTTP).
+- **Jaeger** is configured to receive OTLP traces on port `4317` and exposes
its UI on port `16686`.
+
+The collector forwards all received traces to Jaeger using OTLP/gRPC.
+
+---
+
+## Sending Traces from Impala
+
+To send traces from an Impala cluster to this collector, start Impala with the
following arguments:
+
+```bash
+./bin/start-impala-cluster.py \
+ --cluster_size=2 \
+ --num_coordinators=1 \
+ --use_exclusive_coordinators \
+ --impalad_args="-v=2 --otel_trace_enabled=true \
+ --otel_trace_collector_url=http://localhost:55888/v1/traces
+ --otel_trace_span_processor=simple \
+ --cluster_id=local_cluster"
+```
+
+- Ensure the collector is running before starting Impala.
+- Adjust `--otel_trace_collector_url` if running on a remote host.
+
+---
+
+## Viewing Traces
+
+- Open the Jaeger UI in your browser:
[http://localhost:16686/](http://localhost:16686/)
+- If running remotely, use SSH port forwarding:
+
+ ```bash
+ ssh -L 16686:localhost:16686 <your-dev-machine>
+ ```
+
+---
+
+## Notes
+
+- The scripts and Docker Compose setup are idempotent and safe to run multiple
times.
+- All containers and the custom network are cleaned up on stop.
+- The provided configuration is suitable for local development and testing
only.
+
+---
+
+## Troubleshooting
+
+- **Ports already in use:** Ensure no other services are using ports `55888`,
`4317`, or `16686`.
+- **Containers not starting:** Check Docker logs for `otel-collector` and
`jaeger` for errors:
+
+ ```bash
+ docker logs otel-collector
+ docker logs jaeger
+ ```
+
+- **Configuration changes:** Edit `otel-config.yml` as needed and restart the
services.
+
+---
\ No newline at end of file
diff --git a/testdata/bin/otel-collector/docker-compose.yml
b/testdata/bin/otel-collector/docker-compose.yml
new file mode 100644
index 000000000..d2ab69961
--- /dev/null
+++ b/testdata/bin/otel-collector/docker-compose.yml
@@ -0,0 +1,43 @@
+##############################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+##############################################################################
+
+version: '3.0'
+
+services:
+ jaeger:
+ image: jaegertracing/jaeger:2.5.0
+ container_name: jaeger
+ ports:
+ - "16686:16686" # Jaeger UI
+ - "5778:5778" # Config REST API
+ - "4317:4317" # ingest
+ environment:
+ - COLLECTOR_OTLP_ENABLED=true
+ - COLLECTOR_OTLP_HTTP_ENABLED=true
+
+ otel-collector:
+ image: otel/opentelemetry-collector:latest
+ container_name: otel-collector
+ depends_on:
+ - jaeger
+ volumes:
+ - ./otel-config.yml:/etc/otel/config.yml
+ command: ["--config", "/etc/otel/config.yml"]
+ ports:
+ - "55888:55888" # Expose OTLP HTTP externally
diff --git a/testdata/bin/otel-collector/otel-config.yml
b/testdata/bin/otel-collector/otel-config.yml
new file mode 100644
index 000000000..f15c35172
--- /dev/null
+++ b/testdata/bin/otel-collector/otel-config.yml
@@ -0,0 +1,36 @@
+##############################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+##############################################################################
+
+receivers:
+ otlp:
+ protocols:
+ http:
+ endpoint: 0.0.0.0:55888
+
+exporters:
+ otlp:
+ endpoint: jaeger:4317
+ tls:
+ insecure: true
+
+service:
+ pipelines:
+ traces:
+ receivers: [otlp]
+ exporters: [otlp]
