obelix74 commented on code in PR #3924: URL: https://github.com/apache/polaris/pull/3924#discussion_r2880510422
########## site/content/in-dev/unreleased/proposals/observability-rest-api.md: ########## @@ -0,0 +1,922 @@ +--- +title: Observability REST API +linkTitle: Observability REST API +weight: 100 +--- +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Proposal: REST API for Querying Table Metrics and Events + +**Author:** Anand Sankaran +**Date:** 2026-03-02 +**Status:** Draft Proposal +**Target:** Apache Polaris + +--- + +## Abstract + +This proposal defines REST API endpoints for querying table metrics and catalog events from Apache Polaris. The endpoints expose data already being persisted via the existing JDBC persistence model (`events`, `scan_metrics_report`, `commit_metrics_report` tables) and follow established Polaris API patterns. + +--- + +## Table of Contents + +1. [Motivation](#1-motivation) +2. [Use Cases](#2-use-cases) +3. [Design Principles](#3-design-principles) +4. [API Specification](#4-api-specification) +5. [Authorization](#5-authorization) +6. [OpenAPI Schema](#6-openapi-schema) +7. [Implementation Notes](#7-implementation-notes) + +--- + +## 1. Motivation + +Apache Polaris currently persists table metrics (scan reports, commit reports) and catalog events to the database, but provides no REST API to query this data. Users must access the database directly to retrieve metrics or audit information. + +Adding read-only REST endpoints enables: +- Programmatic access to metrics without database credentials +- Integration with monitoring dashboards and alerting systems +- Consistent authorization via Polaris RBAC +- Pagination and filtering without writing SQL + +--- + +## 2. Use Cases + +### 2.1 Table Health Monitoring +- Track write patterns: files added/removed per commit, record counts, duration trends +- Identify tables with high commit frequency or unusually large commits +- Detect issues indicating need for compaction (many small files) or optimization + +### 2.2 Query Performance Analysis +- Understand read patterns: files scanned vs skipped, planning duration +- Identify inefficient queries with low manifest/file pruning ratios +- Correlate performance with filter expressions and projected columns + +### 2.3 Capacity Planning & Chargeback +- Aggregate metrics by table, namespace, or principal over time +- Track storage growth trends (`total_file_size_bytes`) +- Attribute usage to teams/users via `principal_name` + +### 2.4 Debugging & Troubleshooting +- Correlate metrics with distributed traces (`otel_trace_id`, `otel_span_id`) +- Investigate specific commits by `snapshot_id` +- Trace operations via `request_id` + +### 2.5 Audit & Compliance +- Track who created/dropped/modified catalog objects +- Monitor administrative actions (credential rotation, grant changes) +- Generate compliance reports for access patterns + +--- + +## 3. Design Principles + +| Principle | Rationale | +|-----------|-----------| +| **Management API namespace** | Use `/api/management/v1/...` to separate from Iceberg REST Catalog paths | +| **Read-only endpoints** | Only GET methods; metrics/events are written via existing flows | +| **Consistent pagination** | Follow existing `pageToken`/`nextPageToken` patterns | +| **Flexible filtering** | Time ranges, principal, snapshot - common query patterns | +| **RBAC integration** | Leverage existing Polaris authorization model | + +--- + +## 4. API Specification + +### 4.1 Endpoint Summary + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/management/v1/catalogs/{catalogName}/events` | List events for a catalog | +| GET | `/api/management/v1/catalogs/{catalogName}/events/{eventId}` | Get a specific event | +| GET | `/api/management/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/scan-metrics` | List scan metrics for a table | Review Comment: @dimas-b and @nandorKollar Please see https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?tab=t.0 and https://github.com/apache/iceberg/pull/12584. Looks like the Iceberg events API is ` /v1/{prefix}/events:`. I have updated the markdown to be in sync. Is this ok? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
