obelix74 commented on code in PR #3924: URL: https://github.com/apache/polaris/pull/3924#discussion_r2880536884
########## site/content/in-dev/unreleased/proposals/observability-rest-api.md: ########## @@ -0,0 +1,922 @@ +--- +title: Observability REST API +linkTitle: Observability REST API +weight: 100 +--- +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Proposal: REST API for Querying Table Metrics and Events + +**Author:** Anand Sankaran +**Date:** 2026-03-02 +**Status:** Draft Proposal +**Target:** Apache Polaris + +--- + +## Abstract + +This proposal defines REST API endpoints for querying table metrics and catalog events from Apache Polaris. The endpoints expose data already being persisted via the existing JDBC persistence model (`events`, `scan_metrics_report`, `commit_metrics_report` tables) and follow established Polaris API patterns. + +--- + +## Table of Contents + +1. [Motivation](#1-motivation) +2. [Use Cases](#2-use-cases) +3. [Design Principles](#3-design-principles) +4. [API Specification](#4-api-specification) +5. [Authorization](#5-authorization) +6. [OpenAPI Schema](#6-openapi-schema) +7. [Implementation Notes](#7-implementation-notes) + +--- + +## 1. Motivation + +Apache Polaris currently persists table metrics (scan reports, commit reports) and catalog events to the database, but provides no REST API to query this data. Users must access the database directly to retrieve metrics or audit information. + +Adding read-only REST endpoints enables: +- Programmatic access to metrics without database credentials +- Integration with monitoring dashboards and alerting systems +- Consistent authorization via Polaris RBAC +- Pagination and filtering without writing SQL + +--- + +## 2. Use Cases + +### 2.1 Table Health Monitoring +- Track write patterns: files added/removed per commit, record counts, duration trends +- Identify tables with high commit frequency or unusually large commits +- Detect issues indicating need for compaction (many small files) or optimization + +### 2.2 Query Performance Analysis +- Understand read patterns: files scanned vs skipped, planning duration +- Identify inefficient queries with low manifest/file pruning ratios +- Correlate performance with filter expressions and projected columns + +### 2.3 Capacity Planning & Chargeback +- Aggregate metrics by table, namespace, or principal over time +- Track storage growth trends (`total_file_size_bytes`) +- Attribute usage to teams/users via `principal_name` + +### 2.4 Debugging & Troubleshooting +- Correlate metrics with distributed traces (`otel_trace_id`, `otel_span_id`) +- Investigate specific commits by `snapshot_id` +- Trace operations via `request_id` + +### 2.5 Audit & Compliance +- Track who created/dropped/modified catalog objects +- Monitor administrative actions (credential rotation, grant changes) +- Generate compliance reports for access patterns + +--- + +## 3. Design Principles + +| Principle | Rationale | +|-----------|-----------| +| **Management API namespace** | Use `/api/management/v1/...` to separate from Iceberg REST Catalog paths | +| **Read-only endpoints** | Only GET methods; metrics/events are written via existing flows | +| **Consistent pagination** | Follow existing `pageToken`/`nextPageToken` patterns | +| **Flexible filtering** | Time ranges, principal, snapshot - common query patterns | +| **RBAC integration** | Leverage existing Polaris authorization model | + +--- + +## 4. API Specification + +### 4.1 Endpoint Summary + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/management/v1/catalogs/{catalogName}/events` | List events for a catalog | +| GET | `/api/management/v1/catalogs/{catalogName}/events/{eventId}` | Get a specific event | +| GET | `/api/management/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/scan-metrics` | List scan metrics for a table | +| GET | `/api/management/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/commit-metrics` | List commit metrics for a table | + +### 4.2 Path Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `catalogName` | string | Name of the catalog | +| `namespace` | string | Namespace (URL-encoded, multi-level separated by `%1F`) | +| `table` | string | Table name | +| `eventId` | string | Unique event identifier | + +### 4.3 Query Parameters + +#### List Events (`/catalogs/{catalogName}/events`) + +| Parameter | Type | Required | Default | Description | +|-----------|------|----------|---------|-------------| +| `pageToken` | string | No | - | Cursor for pagination (from previous response) | +| `pageSize` | integer | No | 100 | Results per page (max: 1000) | +| `eventType` | string | No | - | Filter by event type (e.g., `AFTER_CREATE_TABLE`) | +| `resourceType` | string | No | - | Filter by resource: `CATALOG`, `NAMESPACE`, `TABLE`, `VIEW` | +| `resourceIdentifier` | string | No | - | Filter by resource identifier (exact match) | +| `principalName` | string | No | - | Filter by principal who triggered the event | +| `timestampFrom` | long | No | - | Start of time range (epoch milliseconds, inclusive) | +| `timestampTo` | long | No | - | End of time range (epoch milliseconds, exclusive) | + +#### List Scan Metrics (`/.../tables/{table}/scan-metrics`) + +| Parameter | Type | Required | Default | Description | +|-----------|------|----------|---------|-------------| +| `pageToken` | string | No | - | Cursor for pagination | +| `pageSize` | integer | No | 100 | Results per page (max: 1000) | +| `snapshotId` | long | No | - | Filter by snapshot ID | +| `principalName` | string | No | - | Filter by principal | +| `timestampFrom` | long | No | - | Start of time range (epoch ms) | +| `timestampTo` | long | No | - | End of time range (epoch ms) | + +#### List Commit Metrics (`/.../tables/{table}/commit-metrics`) + +| Parameter | Type | Required | Default | Description | +|-----------|------|----------|---------|-------------| +| `pageToken` | string | No | - | Cursor for pagination | +| `pageSize` | integer | No | 100 | Results per page (max: 1000) | +| `snapshotId` | long | No | - | Filter by snapshot ID | +| `operation` | string | No | - | Filter by operation: `append`, `overwrite`, `delete`, `replace` | +| `principalName` | string | No | - | Filter by principal | +| `timestampFrom` | long | No | - | Start of time range (epoch ms) | +| `timestampTo` | long | No | - | End of time range (epoch ms) | + +### 4.4 Example Requests and Responses + +#### List Events + +**Request:** +```http +GET /api/management/v1/catalogs/my-catalog/events?pageSize=2&eventType=AFTER_CREATE_TABLE×tampFrom=1709251200000 +Authorization: Bearer <token> +``` + +**Response:** +```json +{ + "nextPageToken": "eyJ0cyI6MTcwOTMzNzYxMjM0NSwiaWQiOiI1NTBlODQwMCJ9", + "events": [ + { + "eventId": "550e8400-e29b-41d4-a716-446655440000", + "catalogId": "my-catalog", + "requestId": "req-12345", + "eventType": "AFTER_CREATE_TABLE", + "timestampMs": 1709337612345, + "principalName": "[email protected]", + "resourceType": "TABLE", + "resourceIdentifier": "analytics.events.page_views", + "additionalProperties": { + "table-uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890" + } + }, + { + "eventId": "661f9511-f30c-52e5-b827-557766551111", + "catalogId": "my-catalog", + "requestId": "req-12346", + "eventType": "AFTER_CREATE_TABLE", + "timestampMs": 1709337500000, + "principalName": "[email protected]", + "resourceType": "TABLE", + "resourceIdentifier": "analytics.events.user_actions", + "additionalProperties": { + "table-uuid": "b2c3d4e5-f6a7-8901-bcde-f23456789012" + } + } + ] +} +``` + +#### Get Single Event + +**Request:** +```http +GET /api/management/v1/catalogs/my-catalog/events/550e8400-e29b-41d4-a716-446655440000 +Authorization: Bearer <token> +``` + +**Response:** +```json +{ + "eventId": "550e8400-e29b-41d4-a716-446655440000", + "catalogId": "my-catalog", + "requestId": "req-12345", + "eventType": "AFTER_CREATE_TABLE", + "timestampMs": 1709337612345, + "principalName": "[email protected]", + "resourceType": "TABLE", + "resourceIdentifier": "analytics.events.page_views", + "additionalProperties": { + "table-uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "traceparent": "00-abc123def456789012345678901234-def456789012-01" + } +} +``` + +#### List Scan Metrics + +**Request:** +```http +GET /api/management/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views/scan-metrics?pageSize=2×tampFrom=1709251200000 +Authorization: Bearer <token> +``` + +**Response:** +```json +{ + "nextPageToken": null, + "reports": [ + { + "reportId": "scan-001-abc123", + "catalogId": 12345, + "tableId": 67890, + "timestampMs": 1709337612345, + "principalName": "[email protected]", + "requestId": "req-scan-001", + "otelTraceId": "abc123def456789012345678901234", + "otelSpanId": "def456789012", + "snapshotId": 1234567890123, + "schemaId": 0, + "filterExpression": "event_date >= '2024-03-01'", + "projectedFieldIds": "1,2,3,5,8", + "projectedFieldNames": "event_id,user_id,event_type,timestamp,page_url", + "resultDataFiles": 150, + "resultDeleteFiles": 5, + "totalFileSizeBytes": 1073741824, + "totalDataManifests": 12, + "totalDeleteManifests": 2, + "scannedDataManifests": 8, + "scannedDeleteManifests": 2, + "skippedDataManifests": 4, + "skippedDeleteManifests": 0, + "skippedDataFiles": 45, + "skippedDeleteFiles": 0, + "totalPlanningDurationMs": 250, + "equalityDeleteFiles": 3, + "positionalDeleteFiles": 2, + "indexedDeleteFiles": 0, + "totalDeleteFileSizeBytes": 52428800 + } + ] +} +``` + +#### List Commit Metrics + +**Request:** +```http +GET /api/management/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views/commit-metrics?operation=append&pageSize=2 +Authorization: Bearer <token> +``` + +**Response:** +```json +{ + "nextPageToken": "eyJ0cyI6MTcwOTMzNzcwMDAwMCwiaWQiOiJjb21taXQtMDAyIn0=", + "reports": [ + { + "reportId": "commit-001-xyz789", + "catalogId": 12345, + "tableId": 67890, + "timestampMs": 1709337800000, + "principalName": "[email protected]", + "requestId": "req-commit-001", + "otelTraceId": "xyz789abc123456789012345678901", + "otelSpanId": "abc123456789", + "snapshotId": 1234567890124, + "sequenceNumber": 42, + "operation": "append", + "addedDataFiles": 10, + "removedDataFiles": 0, + "totalDataFiles": 160, + "addedDeleteFiles": 0, + "removedDeleteFiles": 0, + "totalDeleteFiles": 5, + "addedEqualityDeleteFiles": 0, + "removedEqualityDeleteFiles": 0, + "addedPositionalDeleteFiles": 0, + "removedPositionalDeleteFiles": 0, + "addedRecords": 100000, + "removedRecords": 0, + "totalRecords": 15000000, + "addedFileSizeBytes": 104857600, + "removedFileSizeBytes": 0, + "totalFileSizeBytes": 1178599424, + "totalDurationMs": 5000, + "attempts": 1 + } + ] +} +``` + +--- + +## 5. Authorization + +### 5.1 Required Privileges + +| Endpoint | Required Privilege | Scope | +|----------|-------------------|-------| +| List/Get Events | `CATALOG_MANAGE_METADATA` | Catalog | +| List Scan Metrics | `TABLE_READ_DATA` | Table | +| List Commit Metrics | `TABLE_READ_DATA` | Table | + +### 5.2 Rationale + +- **Events** contain catalog-wide audit information and should require catalog-level administrative access +- **Metrics** are table-specific and align with read access since they describe query patterns and commit history +- This follows the principle of least privilege while enabling common use cases + +### 5.3 Alternative: New Privileges + +If finer-grained control is desired, new privileges could be introduced: + +| New Privilege | Description | +|---------------|-------------| +| `CATALOG_READ_EVENTS` | Read-only access to catalog events | +| `TABLE_READ_METRICS` | Read-only access to table metrics | + +--- + +## 6. OpenAPI Schema + +Add the following to `spec/polaris-management-service.yml`: + +### 6.1 Paths + +```yaml +paths: + /catalogs/{catalogName}/events: Review Comment: Having it inline helps with review context, but I agree that for implementation it should be in proper spec files: - Events API → spec/rest-catalog-open-api.yaml (extending Iceberg spec) - Metrics API → spec/polaris-management-service.yml Would you prefer I split it out now, or is it acceptable to defer until approval? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
