Ma77Ball commented on code in PR #5377:
URL: https://github.com/apache/texera/pull/5377#discussion_r4358524032
##########
bin/k8s/templates/base/workflow-computing-unit-manager/workflow-computing-unit-manager-deployment.yaml:
##########
@@ -151,4 +151,5 @@ spec:
{{- range .Values.texeraEnvVars }}
- name: {{ .name }}
value: "{{ .value }}"
- {{- end }}
\ No newline at end of file
+ {{- end }}
+ {{- include "texera.observability.env" . | nindent 12 }}
Review Comment:
The manager now forwards its collector endpoint, SDK enable flag, allowed
hosts, and metric interval into each computing unit it creates, and gives each
unit its own service.instance.id (service.name stays shared, any manager
instance id is dropped) so units do not collapse into one metric series. It
forwards nothing when no collector endpoint is set, so a unit stays
default-off. Covered by unit tests on the env-building helper for the enabled,
disabled, and inherited-id cases.
##########
bin/k8s/templates/base/observability/grafana.yaml:
##########
@@ -0,0 +1,152 @@
+{{/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/}}
+
+{{- if and .Values.observability.enabled .Values.observability.grafana.enabled
}}
+# Grafana: dashboard/UI, k8s counterpart of the docker-compose grafana
+# service. Reads the Apache-2.0 backends (Loki, VictoriaMetrics, Jaeger)
+# as pre-provisioned datasources via the ConfigMap below. Grafana is
+# AGPLv3, so it is an optional operator-run component, not part of any
+# Texera release artifact (see bin/observability/README.md). Only this
+# ASF-licensed template lives in the tree. The Service is ClusterIP (no
+# Ingress), matching the rest of the stack; expose it via your own
+# Ingress / port-forward to reach the UI.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: {{ .Release.Name }}-grafana-datasources
+ namespace: {{ .Release.Namespace }}
+ labels:
+ app: {{ .Release.Name }}-grafana
+data:
+ datasources.yaml: |
+ apiVersion: 1
+ datasources:
+ # Metrics: VictoriaMetrics speaks the Prometheus query API. Default.
+ - name: Metrics
+ type: prometheus
+ uid: metrics
+ access: proxy
+ url: http://{{ .Release.Name }}-victoriametrics:{{
.Values.observability.metrics.port }}
+ isDefault: true
+ editable: false
+ {{- if .Values.observability.logs.enabled }}
+ # Logs: Loki, LogQL over OTLP-ingested logs.
+ - name: Loki
+ type: loki
+ uid: loki
+ access: proxy
+ url: http://{{ .Release.Name }}-loki:{{
.Values.observability.logs.port }}
+ editable: false
+ {{- end }}
+ {{- if .Values.observability.traces.enabled }}
+ # Traces: Jaeger query API / UI backend.
+ - name: Jaeger
+ type: jaeger
+ uid: jaeger
+ access: proxy
+ url: http://{{ .Release.Name }}-jaeger:{{
.Values.observability.traces.queryPort }}
+ editable: false
+ {{- end }}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: {{ .Release.Name }}-grafana
+ namespace: {{ .Release.Namespace }}
+ labels:
+ app: {{ .Release.Name }}-grafana
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: {{ .Release.Name }}-grafana
+ template:
+ metadata:
+ labels:
+ app: {{ .Release.Name }}-grafana
+ annotations:
+ # Roll the pod when the provisioned datasources change.
+ checksum/config: {{ include (print $.Template.BasePath
"/base/observability/grafana.yaml") . | sha256sum }}
Review Comment:
Moved the Grafana datasource body and the Loki config body into named
helpers and pointed each checksum/config annotation at its helper, so a
template no longer hashes the file that contains the annotation and helm
template renders with observability enabled. Verified by rendering the
templates.
##########
bin/k8s/templates/base/observability/loki.yaml:
##########
@@ -0,0 +1,168 @@
+{{/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/}}
+
+{{- if and .Values.observability.enabled .Values.observability.logs.enabled }}
+# Loki: log store, k8s counterpart of the docker-compose loki service.
+# OTLP ingest (/otlp) + LogQL query API on one port. Loki is AGPLv3, so
+# it is an optional operator-run component, not part of any Texera release
+# artifact (see bin/observability/README.md). Only this ASF-licensed
+# template lives in the tree.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: {{ .Release.Name }}-loki-config
+ namespace: {{ .Release.Namespace }}
+ labels:
+ app: {{ .Release.Name }}-loki
+data:
+ config.yaml: |
+ # No multi-tenancy: one implicit tenant for the in-cluster stack.
+ auth_enabled: false
+ server:
+ http_listen_port: {{ .Values.observability.logs.port }}
+ grpc_listen_port: 9096
+ common:
+ instance_addr: 127.0.0.1
+ # Every component's data path defaults under this prefix, which is
+ # the PVC mounted at /loki below.
+ path_prefix: /loki
+ storage:
+ filesystem:
+ chunks_directory: /loki/chunks
+ rules_directory: /loki/rules
+ replication_factor: 1
+ ring:
+ kvstore:
+ store: inmemory
+ schema_config:
+ configs:
+ - from: 2024-01-01
+ store: tsdb
+ object_store: filesystem
+ schema: v13
+ index:
+ prefix: index_
+ period: 24h
+ limits_config:
+ # Required for OTLP ingest (OTLP attributes -> structured metadata).
+ allow_structured_metadata: true
+ reject_old_samples: false
+ compactor:
Review Comment:
Wired observability.logs.retentionPeriod into limits_config.retention_period
and turned on the compactor with retention_enabled and delete_request_store so
expired logs are actually deleted, and applied the same settings to the Compose
Loki config.
##########
bin/observability/README.md:
##########
@@ -0,0 +1,131 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Texera Observability Stack
+
+Texera's backend services are instrumented with
[OpenTelemetry](https://opentelemetry.io/)
+and export OTLP (logs, metrics, and traces). This directory is an **optional,
+bundled reference stack** that an operator can start alongside a single-node
+deployment to collect and visualize that telemetry. It is opt-in: nothing here
+runs unless you enable the observability compose profiles (see below), and
+Texera emits no telemetry unless `OTEL_SDK_DISABLED=false` is set.
+
+## Topology
+
+A single OpenTelemetry Collector is the only OTLP ingress. It fans each signal
+out to a dedicated backend, and Grafana sits on top as the query UI:
+
+```
+ Texera services
+ │ OTLP/gRPC (4317) or OTLP/HTTP (4318)
+ ▼
+ otel-collector
+ ┌──────────────┬──────┴────────┬───────────────┐
+ OTLP │ remote-write OTLP │ │ (eBPF agent push)
+ ▼ ▼ ▼ ▼
+ Loki VictoriaMetrics Jaeger Parca
+ (logs) (metrics) (traces) (profiles)
+ │ │ │
+ └──────────────┼────────────────┘
+ ▼
+ Grafana
+ (reads Loki + VictoriaMetrics + Jaeger)
+```
+
+- **Loki** stores logs; the collector exports OTLP to `http://loki:3100/otlp`.
+- **VictoriaMetrics** stores metrics; the collector exports Prometheus
+ remote-write to `http://victoriametrics:8428/api/v1/write`.
+- **Jaeger** stores traces; the collector exports OTLP to `jaeger:4317`.
+- **Parca** stores continuous profiles, pushed by its eBPF agent (not read by
+ Grafana; use the Parca UI on `:7070`).
+- **Grafana** reads Loki, VictoriaMetrics (via the Prometheus query API), and
+ Jaeger as pre-provisioned datasources.
+
+## Starting it
+
+The stack is driven by docker-compose profiles. Enable the signals you want in
+`bin/single-node/.env` via `COMPOSE_PROFILES`, then bring the deployment up:
+
+```
+# bin/single-node/.env
+COMPOSE_PROFILES=observability-collector,observability-logs,observability-metrics,observability-traces,observability-grafana
+```
+
+| Profile | Service(s) |
+|---------------------------|-----------------------------------------|
+| `observability-collector` | otel-collector (OTLP ingress; required) |
+| `observability-logs` | loki |
+| `observability-metrics` | victoriametrics |
+| `observability-traces` | jaeger |
+| `observability-profiles` | parca + privileged eBPF parca-agent |
+| `observability-grafana` | grafana |
+
+Also set `OTEL_SDK_DISABLED=false` so the services actually emit telemetry,
then:
+
+```
+cd bin/single-node && docker compose up -d
+# or the dedicated entry point:
+cd bin/observability && docker compose up -d
+```
+
+Reach **Grafana at http://localhost:3000**. The default login is
+`admin` / `texera` (set via `GRAFANA_ADMIN_USER` / `GRAFANA_ADMIN_PASSWORD` in
+`.env`); change it for anything reachable beyond localhost.
+
+### Pre-provisioned datasources
+
+Grafana comes with three datasources already wired up
Review Comment:
Agreed, dashboards and alerts plus host and JVM metrics are better as a
dedicated follow-up PR so operators get maintained, filterable dashboards
rather than only datasources; this PR keeps the datasources and the licensing
boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
