JackieTien97 opened a new pull request, #17516: URL: https://github.com/apache/iotdb/pull/17516
## Summary This PR adds an **OTLP/HTTP receiver** to IoTDB's REST service, enabling IoTDB to directly ingest [OpenTelemetry](https://opentelemetry.io/) telemetry data (traces, metrics, and logs) from AI coding assistants such as **Claude Code**, **Codex**, **Gemini CLI**, and others. ### Motivation Modern AI coding assistants like Claude Code emit rich telemetry data via the OpenTelemetry protocol — including token usage, API costs, tool execution stats, and session events. Storing this data locally in IoTDB enables developers and teams to: - Analyze token consumption patterns across models (input/output/cacheRead/cacheCreation) - Track API costs per session and over time - Monitor tool execution success rates and latencies - Correlate user prompts with API calls and tool invocations via `prompt.id` ### How It Works The OTLP receiver is embedded in the existing REST service (port 18080) and exposes three standard OTLP/HTTP endpoints: ``` POST /rest/v1/otlp/v1/traces POST /rest/v1/otlp/v1/metrics POST /rest/v1/otlp/v1/logs ``` Both `application/x-protobuf` and `application/json` content types are supported. **Dynamic database routing**: The receiver derives the target database name from each request's `service.name` resource attribute. For example: - `claude-code` → database `claude_code` - `codex` → database `codex` - `Gemini CLI` → database `gemini_cli` Database and tables (`metrics`, `logs`, `traces`) are created automatically on first use. ### Table Schema Design (Table Model) **`metrics` table** — one row per data point, with OTLP attributes extracted into typed columns: | Column | Category | Description | |--------|----------|-------------| | `user_id` | TAG | Anonymous device identifier | | `session_id` | TAG | Session UUID | | `metric_name` | TAG | e.g. `claude_code.token.usage`, `claude_code.cost.usage` | | `model` | TAG | Model name, e.g. `claude-opus-4-7` | | `type` | TAG | Metric sub-type: `input`/`output`/`cacheRead`/`cacheCreation`/`user`/`cli` | | `terminal_type` | ATTRIBUTE | `iTerm.app` / `vscode` / `cursor` | | `service_version` | ATTRIBUTE | Claude Code version | | `os_type`, `os_version`, `host_arch` | ATTRIBUTE | Platform info | | `unit`, `metric_type`, `description` | ATTRIBUTE | Metric metadata | | `value` | FIELD | The metric value (tokens count / USD / seconds) | **`logs` table** — flattened event records with event-type-specific columns: | Column | Category | Description | |--------|----------|-------------| | `user_id`, `session_id` | TAG | Identity | | `event_name` | TAG | `user_prompt` / `api_request` / `api_error` / `tool_result` / `tool_decision` | | `terminal_type`, `service_version`, `os_type`, `host_arch` | ATTRIBUTE | Static per-session | | `prompt_id`, `event_sequence`, `body` | FIELD | Common event fields | | `model`, `cost_usd`, `duration_ms`, `input_tokens`, `output_tokens`, ... | FIELD | API request fields | | `tool_name`, `success`, `tool_duration_ms`, `decision`, `decision_source` | FIELD | Tool fields | **`traces` table** — standard span data with resource attributes extracted. ### How to Use with Claude Code 1. Enable REST service in `iotdb-system.properties`: ```properties enable_rest_service=true rest_service_port=18080 ``` 2. Add the following environment variables to your shell profile (e.g. `~/.zshrc` or `~/.bashrc`): ```bash export CLAUDE_CODE_ENABLE_TELEMETRY=1 export OTEL_METRICS_EXPORTER=otlp export OTEL_LOGS_EXPORTER=otlp export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:18080/rest/v1/otlp ``` 3. Start IoTDB, then start a new Claude Code session. Telemetry data will flow automatically. 4. Query your data: ```sql -- Token usage by model and type USE claude_code; SELECT model, type, sum(value) FROM metrics WHERE metric_name = 'claude_code.token.usage' GROUP BY model, type; -- API cost trend SELECT date_bin(1h, time) AS hour, sum(value) AS cost FROM metrics WHERE metric_name = 'claude_code.cost.usage' GROUP BY 1 ORDER BY 1; -- Tool success rate SELECT tool_name, count_if(success = 'true') AS ok, count(*) AS total FROM logs WHERE event_name = 'tool_result' GROUP BY tool_name; -- Trace a single prompt SELECT * FROM logs WHERE prompt_id = 'xxx' ORDER BY event_sequence; ``` ### Files Changed **New files** (11 files in `external-service-impl/rest/.../protocol/otlp/v1/`): - `OtlpTracesResource`, `OtlpMetricsResource`, `OtlpLogsResource` — JAX-RS endpoints - `OtlpService` — per-database session management + schema init - `OtlpSchemaInitializer` — idempotent `CREATE DATABASE/TABLE IF NOT EXISTS` - `OtlpIngestor` + `OtlpTableBatch` — column-major batch → `InsertTabletStatement` - `OtlpTracesConverter`, `OtlpMetricsConverter`, `OtlpLogsConverter` — OTLP→rows - `OtlpConverter` — timestamp precision, hex encoding, attribute extraction - `OtlpHttp` — protobuf/JSON parsing and response rendering **Modified files** (5 files): - `rest/pom.xml` — added `io.opentelemetry.proto:opentelemetry-proto` + `protobuf-java-util` - `AuthorizationFilter` — bypass auth for `/rest/v1/otlp/` paths (OTLP receiver authenticates internally via `otlp_username/otlp_password`) - `IoTDBRestServiceConfig/Descriptor` — added `otlp_username`/`otlp_password` config - `iotdb-system.properties.template` — documented OTLP configuration section ## Test plan - [x] Build: `mvn clean package -pl distribution -am -DskipTests` succeeds - [x] Smoke test: `curl -X POST http://localhost:18080/rest/v1/otlp/v1/logs` with JSON body returns HTTP 200 - [x] Verify metrics: token.usage with type=input/output creates separate rows in `claude_code.metrics` - [x] Verify logs: api_request and tool_result events land with all fields extracted - [x] Multi-database: requests from different service.name values create separate databases - [x] End-to-end: real Claude Code telemetry data flows into IoTDB tables 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
