zuozhiw commented on code in PR #5377:
URL: https://github.com/apache/texera/pull/5377#discussion_r4346774161
##########
bin/k8s/templates/base/workflow-computing-unit-manager/workflow-computing-unit-manager-deployment.yaml:
##########
@@ -151,4 +151,5 @@ spec:
{{- range .Values.texeraEnvVars }}
- name: {{ .name }}
value: "{{ .value }}"
- {{- end }}
\ No newline at end of file
+ {{- end }}
+ {{- include "texera.observability.env" . | nindent 12 }}
Review Comment:
These settings reach the manager, but it creates CU pods from a separate,
explicit environment map.
`ComputingUnitManagingResource.computingUnitEnvironmentVariables` does not
forward observability settings, and `KubernetesClient` uses that map for the
container environment. From code inspection, new CUs therefore retain the
SDK-disabled/default-endpoint settings even when telemetry is enabled on the
manager. I have not deployed a CU to verify this.
Could we pass the SDK enable/disable setting, OTLP endpoint, allowed hosts,
metric interval, and deployment resource attributes into created CUs? Each CU
needs its own instance ID rather than the manager's. A unit test of the
environment-building helper with telemetry enabled/disabled should cover this
without Kubernetes.
##########
bin/k8s/templates/base/observability/grafana.yaml:
##########
@@ -0,0 +1,152 @@
+{{/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/}}
+
+{{- if and .Values.observability.enabled .Values.observability.grafana.enabled
}}
+# Grafana: dashboard/UI, k8s counterpart of the docker-compose grafana
+# service. Reads the Apache-2.0 backends (Loki, VictoriaMetrics, Jaeger)
+# as pre-provisioned datasources via the ConfigMap below. Grafana is
+# AGPLv3, so it is an optional operator-run component, not part of any
+# Texera release artifact (see bin/observability/README.md). Only this
+# ASF-licensed template lives in the tree. The Service is ClusterIP (no
+# Ingress), matching the rest of the stack; expose it via your own
+# Ingress / port-forward to reach the UI.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: {{ .Release.Name }}-grafana-datasources
+ namespace: {{ .Release.Namespace }}
+ labels:
+ app: {{ .Release.Name }}-grafana
+data:
+ datasources.yaml: |
+ apiVersion: 1
+ datasources:
+ # Metrics: VictoriaMetrics speaks the Prometheus query API. Default.
+ - name: Metrics
+ type: prometheus
+ uid: metrics
+ access: proxy
+ url: http://{{ .Release.Name }}-victoriametrics:{{
.Values.observability.metrics.port }}
+ isDefault: true
+ editable: false
+ {{- if .Values.observability.logs.enabled }}
+ # Logs: Loki, LogQL over OTLP-ingested logs.
+ - name: Loki
+ type: loki
+ uid: loki
+ access: proxy
+ url: http://{{ .Release.Name }}-loki:{{
.Values.observability.logs.port }}
+ editable: false
+ {{- end }}
+ {{- if .Values.observability.traces.enabled }}
+ # Traces: Jaeger query API / UI backend.
+ - name: Jaeger
+ type: jaeger
+ uid: jaeger
+ access: proxy
+ url: http://{{ .Release.Name }}-jaeger:{{
.Values.observability.traces.queryPort }}
+ editable: false
+ {{- end }}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: {{ .Release.Name }}-grafana
+ namespace: {{ .Release.Namespace }}
+ labels:
+ app: {{ .Release.Name }}-grafana
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: {{ .Release.Name }}-grafana
+ template:
+ metadata:
+ labels:
+ app: {{ .Release.Name }}-grafana
+ annotations:
+ # Roll the pod when the provisioned datasources change.
+ checksum/config: {{ include (print $.Template.BasePath
"/base/observability/grafana.yaml") . | sha256sum }}
Review Comment:
This checksum includes the file containing the checksum itself, causing
recursive template rendering. Loki has the same issue at its `checksum/config`
annotation.
Rendering the exact templates with observability enabled reproduced the
error. Disabling Loki still fails in Grafana; disabling observability entirely
passes.
Could we move each ConfigMap/config body into a separate template or helper
and hash only that content? Please add a Helm render check with observability
enabled, since the default-off configuration hides this.
##########
bin/single-node/docker-compose.override.yml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Local-dev override, auto-loaded by `docker compose` when run from this
+# directory. It publishes the OTel collector's OTLP receiver ports on the
+# host loopback so a backend running outside docker (sbt / IntelliJ) can
+# emit telemetry to the collector. The released bundle does not ship this
+# file, so the base compose stays network-clean in a deploy.
+name: texera-single-node
+
+services:
+ otel-collector:
+ ports:
Review Comment:
The canonical `bin/local-dev.sh` path does not load this override.
`bin/local-dev/main.sh` explicitly selects `bin/single-node/docker-compose.yml`
plus `bin/local-dev/docker-compose.override.yml`. Rendering that exact
combination with the collector enabled confirms that it publishes no collector
ports, so a host JVM cannot reach the collector at its loopback OTLP endpoint.
Could we put these port mappings in the local-dev override that the launcher
uses, or explicitly include this file there? Please cover the launcher's exact
Compose file combination in a small configuration test.
##########
bin/k8s/templates/base/observability/loki.yaml:
##########
@@ -0,0 +1,168 @@
+{{/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/}}
+
+{{- if and .Values.observability.enabled .Values.observability.logs.enabled }}
+# Loki: log store, k8s counterpart of the docker-compose loki service.
+# OTLP ingest (/otlp) + LogQL query API on one port. Loki is AGPLv3, so
+# it is an optional operator-run component, not part of any Texera release
+# artifact (see bin/observability/README.md). Only this ASF-licensed
+# template lives in the tree.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: {{ .Release.Name }}-loki-config
+ namespace: {{ .Release.Namespace }}
+ labels:
+ app: {{ .Release.Name }}-loki
+data:
+ config.yaml: |
+ # No multi-tenancy: one implicit tenant for the in-cluster stack.
+ auth_enabled: false
+ server:
+ http_listen_port: {{ .Values.observability.logs.port }}
+ grpc_listen_port: 9096
+ common:
+ instance_addr: 127.0.0.1
+ # Every component's data path defaults under this prefix, which is
+ # the PVC mounted at /loki below.
+ path_prefix: /loki
+ storage:
+ filesystem:
+ chunks_directory: /loki/chunks
+ rules_directory: /loki/rules
+ replication_factor: 1
+ ring:
+ kvstore:
+ store: inmemory
+ schema_config:
+ configs:
+ - from: 2024-01-01
+ store: tsdb
+ object_store: filesystem
+ schema: v13
+ index:
+ prefix: index_
+ period: 24h
+ limits_config:
+ # Required for OTLP ingest (OTLP attributes -> structured metadata).
+ allow_structured_metadata: true
+ reject_old_samples: false
+ compactor:
Review Comment:
The switch to Loki drops the configured retention policy.
`observability.logs.retentionPeriod` still defaults to `30d`, but this template
never uses it. The Compose Loki config also lacks retention settings. Loki's
retention cleanup is disabled by default, so logs accumulate indefinitely.
Could we use that value for `limits_config.retention_period` and enable the
compactor to delete expired logs (`compactor.retention_enabled`), including the
required `compactor.delete_request_store` setting? Please do the equivalent for
Compose. Checking the generated settings is enough; there is no need to wait 30
days. This finding is based on the configs and [Loki's documented
defaults](https://grafana.com/docs/loki/latest/operations/storage/retention/).
##########
bin/single-node/docker-compose.yml:
##########
@@ -590,6 +590,227 @@ services:
command: >
sh -c 'apk add --no-cache curl jq bash > /dev/null 2>&1 && bash
/examples/load-examples.sh'
+ # ========================================================================
+ # Observability stack: collector + logs, metrics, traces, profiles.
+ #
+ # Each signal is a separate compose profile; enable them via
+ # COMPOSE_PROFILES in bin/single-node/.env. Every port binds to
+ # loopback (127.0.0.1) or stays on the bridge network; no 0.0.0.0
+ # host bindings.
+ # ========================================================================
+
+ # OpenTelemetry Collector, the single OTLP ingress for all three
+ # signals. Reads bin/observability/otel-collector/config.yaml.
+ # Apache-2.0; pinned to the contrib distribution.
+ otel-collector:
+ image: otel/opentelemetry-collector-contrib:0.153.0
+ container_name: texera-otel-collector
+ profiles: [observability-collector]
+ restart: always
+ user: "10001:10001"
+ read_only: true
+ security_opt:
+ - no-new-privileges:true
+ volumes:
+ -
../observability/otel-collector/config.yaml:/etc/otelcol-contrib/config.yaml:ro,z
+ command: ["--config=/etc/otelcol-contrib/config.yaml"]
+ # In-compose services reach the OTLP receivers over the bridge
+ # network as otel-collector:4317/:4318, so the base file publishes
+ # no host ports. For a backend running outside docker (sbt /
+ # IntelliJ), docker-compose.override.yml publishes them on the host
+ # loopback in local dev; the released bundle omits that override.
+ deploy:
+ resources:
+ limits:
+ memory: 768M
+ cpus: "1.0"
+
+ # Loki (AGPLv3) log store. OTLP ingest at /otlp, LogQL query API on
+ # 3100. AGPLv3, so it is an optional operator-run component and is NOT
+ # shipped in any Texera release artifact (see bin/observability/README.md).
+ # Reads bin/observability/loki/config.yaml.
+ loki:
+ image: grafana/loki:3.2.0
+ container_name: texera-loki
+ profiles: [observability-logs]
+ restart: always
+ user: "10001:10001"
+ read_only: true
+ security_opt:
+ - no-new-privileges:true
+ volumes:
+ - ../observability/loki/config.yaml:/etc/loki/config.yaml:ro,z
+ - loki_data:/loki
+ # Read-only root filesystem, so Loki's scratch space is an in-memory
+ # tmpfs; all durable state lives on the loki_data volume under /loki.
+ tmpfs:
+ - /tmp
+ command: ["-config.file=/etc/loki/config.yaml"]
+ # Loopback-only host binding so an operator can curl the query
+ # API from the host for ad-hoc debugging without exposing it to
+ # the network.
+ ports:
+ - "127.0.0.1:3100:3100"
+ deploy:
+ resources:
+ limits:
+ memory: 1G
+ cpus: "1.0"
+
+ # Grafana (AGPLv3) dashboard/UI. Reads the Apache-2.0 backends below
+ # (Loki, VictoriaMetrics, Jaeger) as pre-provisioned datasources. The UI
+ # is meant to be opened by a human, but like the backends its port is
+ # bound to loopback only (http://localhost:3000). AGPLv3, so it is an
+ # optional operator-run component and is NOT shipped in any Texera
+ # release artifact (see bin/observability/README.md).
+ grafana:
+ image: grafana/grafana:11.2.0
+ container_name: texera-grafana
+ profiles: [observability-grafana]
+ restart: always
+ security_opt:
+ - no-new-privileges:true
+ depends_on:
+ - loki
+ - victoriametrics
+ - jaeger
+ environment:
+ # Admin login. Defaults are weak dev defaults; override in .env for
+ # any deployment reachable beyond localhost.
+ - GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER:-admin}
+ - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD:-texera}
+ # No anonymous access: the UI requires the admin login above.
+ - GF_AUTH_ANONYMOUS_ENABLED=false
+ # Users cannot self-register.
+ - GF_USERS_ALLOW_SIGN_UP=false
+ volumes:
+ # Datasource provisioning (Loki + VictoriaMetrics + Jaeger), read-only.
+ - ../observability/grafana/provisioning:/etc/grafana/provisioning:ro,z
+ - grafana_data:/var/lib/grafana
+ # Loopback-only host binding, like the backends: a human opens
+ # http://localhost:3000, and the UI is never reachable from the
+ # network by default.
+ ports:
+ - "127.0.0.1:${GRAFANA_PORT:-3000}:3000"
+ deploy:
+ resources:
+ limits:
+ memory: 512M
+ cpus: "0.5"
+
+ # VictoriaMetrics, Apache-2.0 metrics store. Accepts Prometheus
+ # remote-write from the collector at /api/v1/write, MetricsQL on
+ # /api/v1/query.
+ victoriametrics:
+ image: victoriametrics/victoria-metrics:v1.144.0
+ container_name: texera-victoriametrics
+ profiles: [observability-metrics]
+ restart: always
+ user: "10003:10003"
Review Comment:
VictoriaMetrics cannot write to a fresh `victoriametrics_data` volume with
this UID/GID. Starting the pinned image with these settings exited with `cannot
create lock file "/data/flock.lock": ... permission denied`.
Could we initialize the volume ownership for UID/GID 10003, or use a
writable data-directory/user arrangement supported by the image? Keep the
service non-root and its root filesystem read-only; `/data` needs to be
writable. Please include a startup check with a fresh volume, since an already
initialized volume can hide this.
##########
bin/observability/README.md:
##########
@@ -0,0 +1,131 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Texera Observability Stack
+
+Texera's backend services are instrumented with
[OpenTelemetry](https://opentelemetry.io/)
+and export OTLP (logs, metrics, and traces). This directory is an **optional,
+bundled reference stack** that an operator can start alongside a single-node
+deployment to collect and visualize that telemetry. It is opt-in: nothing here
+runs unless you enable the observability compose profiles (see below), and
+Texera emits no telemetry unless `OTEL_SDK_DISABLED=false` is set.
+
+## Topology
+
+A single OpenTelemetry Collector is the only OTLP ingress. It fans each signal
+out to a dedicated backend, and Grafana sits on top as the query UI:
+
+```
+ Texera services
+ │ OTLP/gRPC (4317) or OTLP/HTTP (4318)
+ ▼
+ otel-collector
+ ┌──────────────┬──────┴────────┬───────────────┐
+ OTLP │ remote-write OTLP │ │ (eBPF agent push)
+ ▼ ▼ ▼ ▼
+ Loki VictoriaMetrics Jaeger Parca
+ (logs) (metrics) (traces) (profiles)
+ │ │ │
+ └──────────────┼────────────────┘
+ ▼
+ Grafana
+ (reads Loki + VictoriaMetrics + Jaeger)
+```
+
+- **Loki** stores logs; the collector exports OTLP to `http://loki:3100/otlp`.
+- **VictoriaMetrics** stores metrics; the collector exports Prometheus
+ remote-write to `http://victoriametrics:8428/api/v1/write`.
+- **Jaeger** stores traces; the collector exports OTLP to `jaeger:4317`.
+- **Parca** stores continuous profiles, pushed by its eBPF agent (not read by
+ Grafana; use the Parca UI on `:7070`).
+- **Grafana** reads Loki, VictoriaMetrics (via the Prometheus query API), and
+ Jaeger as pre-provisioned datasources.
+
+## Starting it
+
+The stack is driven by docker-compose profiles. Enable the signals you want in
+`bin/single-node/.env` via `COMPOSE_PROFILES`, then bring the deployment up:
+
+```
+# bin/single-node/.env
+COMPOSE_PROFILES=observability-collector,observability-logs,observability-metrics,observability-traces,observability-grafana
+```
+
+| Profile | Service(s) |
+|---------------------------|-----------------------------------------|
+| `observability-collector` | otel-collector (OTLP ingress; required) |
+| `observability-logs` | loki |
+| `observability-metrics` | victoriametrics |
+| `observability-traces` | jaeger |
+| `observability-profiles` | parca + privileged eBPF parca-agent |
+| `observability-grafana` | grafana |
+
+Also set `OTEL_SDK_DISABLED=false` so the services actually emit telemetry,
then:
+
+```
+cd bin/single-node && docker compose up -d
+# or the dedicated entry point:
+cd bin/observability && docker compose up -d
+```
+
+Reach **Grafana at http://localhost:3000**. The default login is
+`admin` / `texera` (set via `GRAFANA_ADMIN_USER` / `GRAFANA_ADMIN_PASSWORD` in
+`.env`); change it for anything reachable beyond localhost.
+
+### Pre-provisioned datasources
+
+Grafana comes with three datasources already wired up
Review Comment:
Could we add operational observability in a follow-up PR? The configured
Grafana datasources are useful. We also need:
- Host CPU/memory and JVM heap/GC/thread metrics.
- Maintained dashboards for workflow counts, failures, and latency, with
environment/service/instance filters.
- A small set of example alerts.
The OTel Collector's host-metrics and Kubernetes kubelet-stats receivers can
collect infrastructure metrics; Texera should expose application/runtime
metrics. Database metrics can come later.
Texera-authored dashboards and configuration can live in this repository
under its Apache license. Keep backends replaceable and Grafana/Loki optional,
following the licensing boundary documented here. This is a follow-up request;
it does not all need to be added in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
