adutra commented on code in PR #3924: URL: https://github.com/apache/polaris/pull/3924#discussion_r2931628291
########## site/content/in-dev/unreleased/proposals/observability-rest-api.md: ########## @@ -0,0 +1,1403 @@ +--- +title: Observability REST API +linkTitle: Observability REST API +weight: 100 +--- +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Proposal: REST API for Querying Table Metrics and Events + +**Author:** Anand Sankaran +**Date:** 2026-03-02 +**Status:** Draft Proposal +**Target:** Apache Polaris + +--- + +## Abstract + +This proposal defines REST API endpoints for querying table metrics and catalog events from Apache Polaris. The endpoints expose data already being persisted via the existing JDBC persistence model (`events`, `scan_metrics_report`, `commit_metrics_report` tables) and follow established Polaris API patterns. + +**Note:** The Events API in this proposal is designed to align with the emerging [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584), which is nearing consensus in the Apache Iceberg community. This ensures forward compatibility and consistency with the broader Iceberg ecosystem. + +--- + +## Table of Contents + +1. [Motivation](#1-motivation) +2. [Use Cases](#2-use-cases) +3. [Design Principles](#3-design-principles) +4. [API Specification](#4-api-specification) +5. [Authorization](#5-authorization) +6. [OpenAPI Schema](#6-openapi-schema) +7. [Implementation Notes](#7-implementation-notes) +8. [Iceberg Events API Alignment](#8-iceberg-events-api-alignment) + +--- + +## 1. Motivation + +Apache Polaris currently persists table metrics (scan reports, commit reports) and catalog events to the database, but provides no REST API to query this data. Users must access the database directly to retrieve metrics or audit information. + +Adding read-only REST endpoints enables: +- Programmatic access to metrics without database credentials +- Integration with monitoring dashboards and alerting systems +- Consistent authorization via Polaris RBAC +- Pagination and filtering without writing SQL + +--- + +## 2. Use Cases + +### 2.1 Table Health Monitoring +- Track write patterns: files added/removed per commit, record counts, duration trends +- Identify tables with high commit frequency or unusually large commits +- Detect issues indicating need for compaction (many small files) or optimization + +### 2.2 Query Performance Analysis +- Understand read patterns: files scanned vs skipped, planning duration +- Identify inefficient queries with low manifest/file pruning ratios +- Correlate performance with filter expressions and projected columns + +### 2.3 Capacity Planning & Chargeback +- Aggregate metrics by table, namespace, or principal over time +- Track storage growth trends (`total_file_size_bytes`) +- Attribute usage to teams/users via `principal_name` + +### 2.4 Debugging & Troubleshooting +- Correlate metrics with distributed traces (`otel_trace_id`, `otel_span_id`) +- Investigate specific commits by `snapshot_id` +- Trace operations via `request_id` + +### 2.5 Audit & Compliance +- Track who created/dropped/modified catalog objects +- Monitor administrative actions (credential rotation, grant changes) +- Generate compliance reports for access patterns + +--- + +## 3. Design Principles + +| Principle | Rationale | +|-----------|-----------| +| **Iceberg Events API alignment** | Events API follows the [Iceberg Events API spec](https://github.com/apache/iceberg/pull/12584) for ecosystem compatibility | +| **Dedicated metrics-reports namespace** | Metrics APIs use `/api/metrics-reports/v1/...` to separate from management and catalog APIs | +| **POST for complex filtering** | Events API uses POST with request body (per Iceberg spec) to support complex filters (arrays, nested objects) | +| **Read-only semantics** | All endpoints are read-only; metrics/events are written via existing flows | +| **Consistent pagination** | Follow `continuation-token` pattern (Iceberg) and `pageToken` pattern (Polaris APIs) | +| **Flexible filtering** | Time ranges, operation types, catalog objects - common query patterns | +| **RBAC integration** | Leverage existing Polaris authorization model | +| **Realm handling** | Process Polaris realms consistently with existing APIs; realm context is derived from the authenticated principal | + +--- + +## 4. API Specification + +### 4.1 Endpoint Summary + +| Method | Path | Description | +|--------|------|-------------| +| POST | `/api/events/v1/{prefix}` | Query events for a catalog | +| GET | `/api/metrics-reports/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}` | List metrics for a table (type specified via query parameter) | + +> **Note:** The Events API uses a dedicated `/api/events/v1/` namespace to avoid URI path clashes with the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584) until that spec is approved. The API design follows Iceberg Events API patterns for future compatibility. The metrics API uses a dedicated `/api/metrics-reports/v1/` namespace since it exposes pre-populated records rather than managing catalog state. + +### 4.2 Path Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `prefix` | string | Catalog prefix (typically the catalog name) | +| `catalogName` | string | Name of the catalog | +| `namespace` | string | Namespace (URL-encoded, multi-level separated by `%1F`) | +| `table` | string | Table name | + +### 4.3 Events API (Iceberg-Compatible) + +The Events API follows the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584) for ecosystem compatibility. Key design decisions from the Iceberg spec: + +- **POST method**: Allows complex filtering with arrays and nested objects in the request body +- **Continuation token**: Opaque cursor for resumable pagination +- **Operation-centric model**: Events are structured around operations (create-table, update-table, etc.) +- **Custom extensions**: Support for `polaris-` prefixed custom operation types for Polaris-specific events + +#### Request Body (`QueryEventsRequest`) + +> **Note:** This request schema matches the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584) for ecosystem compatibility. + +| Property | Type | Required | Description | +|----------|------|----------|-------------| +| `continuation-token` | string | No | Opaque cursor to resume fetching from previous request | +| `page-size` | integer | No | Maximum events per page (server may return fewer) | +| `after-timestamp-ms` | long | No | Filter: events after this timestamp (inclusive) | +| `operation-types` | array[string] | No | Filter by operation types (see below) | +| `catalog-objects-by-name` | array[array[string]] | No | Filter by namespace/table/view names | +| `catalog-objects-by-id` | array[object] | No | Filter by table/view UUIDs | +| `object-types` | array[string] | No | Filter by object type: `namespace`, `table`, `view` | +| `custom-filters` | object | No | Implementation-specific filter extensions | + +#### Standard Operation Types + +| Operation Type | Description | +|----------------|-------------| +| `create-table` | Table created and committed | +| `register-table` | Existing table registered in catalog | +| `drop-table` | Table dropped | +| `update-table` | Table metadata updated | +| `rename-table` | Table renamed | +| `create-view` | View created | +| `drop-view` | View dropped | +| `update-view` | View updated | +| `rename-view` | View renamed | +| `create-namespace` | Namespace created | +| `update-namespace-properties` | Namespace properties updated | +| `drop-namespace` | Namespace dropped | + +#### Polaris Custom Operation Types + +For Polaris-specific events not covered by the Iceberg spec, use the `polaris-` prefix: + +| Custom Operation Type | Description | +|----------------------|-------------| +| `polaris-create-catalog-role` | Catalog role created | +| `polaris-grant-privilege` | Privilege granted | +| `polaris-rotate-credentials` | Principal credentials rotated | +| `polaris-create-policy` | Policy created | +| `polaris-attach-policy` | Policy attached to resource | + +### 4.4 Query Parameters (Metrics API) + +#### List Table Metrics (`/.../tables/{table}`) + +| Parameter | Type | Required | Default | Description | +|-----------|------|----------|---------|-------------| +| `metricType` | string | **Yes** | - | Type of metrics to retrieve: `scan` or `commit` | +| `pageToken` | string | No | - | Cursor for pagination | +| `pageSize` | integer | No | 100 | Results per page (max: 1000) | +| `snapshotId` | long | No | - | Filter by snapshot ID | +| `principalName` | string | No | - | Filter by principal | +| `timestampFrom` | long | No | - | Start of time range (epoch ms) | +| `timestampTo` | long | No | - | End of time range (epoch ms) | +| `operation` | string | No | - | Filter by commit operation (only applicable when `metricType=commit`): `append`, `overwrite`, `delete`, `replace` | + +> **Note:** The `metricType` parameter is required. This design allows for future extensibility as new metric types are added (e.g., compaction metrics, maintenance metrics) without requiring new endpoints. + +### 4.5 Example Requests and Responses + +#### Query Events (Iceberg-Compatible) + +**Request:** +```http +POST /api/events/v1/my-catalog +Authorization: Bearer <token> +Content-Type: application/json + +{ + "page-size": 2, + "operation-types": ["create-table", "update-table"], + "after-timestamp-ms": 1709251200000, + "catalog-objects-by-name": [ + ["analytics", "events"] + ], + "object-types": ["table"] +} +``` + +**Response:** +```json +{ + "next-page-token": "eyJ0cyI6MTcwOTMzNzYxMjM0NSwiaWQiOiI1NTBlODQwMCJ9", + "highest-processed-timestamp-ms": 1709337612345, + "events": [ + { + "event-id": "550e8400-e29b-41d4-a716-446655440000", + "request-id": "req-12345", + "request-event-count": 1, + "timestamp-ms": 1709337612345, + "actor": { + "principal": "[email protected]", + "client-ip": "192.168.1.100" + }, + "operation": { + "operation-type": "create-table", + "identifier": { + "namespace": ["analytics", "events"], + "name": "page_views" + }, + "table-uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "updates": [ + {"action": "assign-uuid", "uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"}, + {"action": "set-current-schema", "schema-id": 0}, + {"action": "set-default-spec", "spec-id": 0} + ] + } + }, + { + "event-id": "661f9511-f30c-52e5-b827-557766551111", + "request-id": "req-12346", + "request-event-count": 1, + "timestamp-ms": 1709337500000, + "actor": { + "principal": "[email protected]" + }, + "operation": { + "operation-type": "update-table", + "identifier": { + "namespace": ["analytics", "events"], + "name": "user_actions" + }, + "table-uuid": "b2c3d4e5-f6a7-8901-bcde-f23456789012", + "updates": [ + {"action": "add-snapshot", "snapshot-id": 123456789} + ], + "requirements": [ + {"type": "assert-table-uuid", "uuid": "b2c3d4e5-f6a7-8901-bcde-f23456789012"} + ] + } + } + ] +} +``` + +#### Query Events with Custom Polaris Operations + +**Request:** +```http +POST /api/events/v1/my-catalog +Authorization: Bearer <token> +Content-Type: application/json + +{ + "page-size": 10, + "operation-types": ["polaris-grant-privilege", "polaris-rotate-credentials"] +} +``` + +**Response:** +```json +{ + "next-page-token": "eyJ0cyI6MTcwOTMzODAwMDAwMH0=", + "highest-processed-timestamp-ms": 1709338000000, + "events": [ + { + "event-id": "772f0622-g41d-63f6-c938-668877662222", + "request-id": "req-admin-001", + "request-event-count": 1, + "timestamp-ms": 1709338000000, + "actor": { + "principal": "[email protected]" + }, + "operation": { + "operation-type": "custom", + "custom-type": "polaris-grant-privilege", + "identifier": { + "namespace": ["analytics", "events"], + "name": "page_views" + }, + "table-uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "privilege": "TABLE_READ_DATA", + "grantee": "data-analyst-role" + } + } + ] +} +``` + +#### List Metrics (Scan) + +**Request:** +```http +GET /api/metrics-reports/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views?metricType=scan&pageSize=2×tampFrom=1709251200000 +Authorization: Bearer <token> +``` + +**Response:** +```json +{ + "nextPageToken": null, + "metricType": "scan", + "reports": [ + { + "reportId": "scan-001-abc123", + "catalogId": 12345, + "tableId": 67890, + "timestampMs": 1709337612345, + "principalName": "[email protected]", + "requestId": "req-scan-001", + "otelTraceId": "abc123def456789012345678901234", + "otelSpanId": "def456789012", + "snapshotId": 1234567890123, + "schemaId": 0, + "filterExpression": "event_date >= '2024-03-01'", + "projectedFieldIds": "1,2,3,5,8", + "projectedFieldNames": "event_id,user_id,event_type,timestamp,page_url", + "resultDataFiles": 150, + "resultDeleteFiles": 5, + "totalFileSizeBytes": 1073741824, + "totalDataManifests": 12, + "totalDeleteManifests": 2, + "scannedDataManifests": 8, + "scannedDeleteManifests": 2, + "skippedDataManifests": 4, + "skippedDeleteManifests": 0, + "skippedDataFiles": 45, + "skippedDeleteFiles": 0, + "totalPlanningDurationMs": 250, + "equalityDeleteFiles": 3, + "positionalDeleteFiles": 2, + "indexedDeleteFiles": 0, + "totalDeleteFileSizeBytes": 52428800 + } + ] +} +``` + +#### List Metrics (Commit) + +**Request:** +```http +GET /api/metrics-reports/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views?metricType=commit&operation=append&pageSize=2 +Authorization: Bearer <token> +``` + +**Response:** +```json +{ + "nextPageToken": "eyJ0cyI6MTcwOTMzNzcwMDAwMCwiaWQiOiJjb21taXQtMDAyIn0=", + "metricType": "commit", + "reports": [ + { + "reportId": "commit-001-xyz789", + "catalogId": 12345, + "tableId": 67890, + "timestampMs": 1709337800000, + "principalName": "[email protected]", + "requestId": "req-commit-001", + "otelTraceId": "xyz789abc123456789012345678901", + "otelSpanId": "abc123456789", + "snapshotId": 1234567890124, + "sequenceNumber": 42, + "operation": "append", + "addedDataFiles": 10, + "removedDataFiles": 0, + "totalDataFiles": 160, + "addedDeleteFiles": 0, + "removedDeleteFiles": 0, + "totalDeleteFiles": 5, + "addedEqualityDeleteFiles": 0, + "removedEqualityDeleteFiles": 0, + "addedPositionalDeleteFiles": 0, + "removedPositionalDeleteFiles": 0, + "addedRecords": 100000, + "removedRecords": 0, + "totalRecords": 15000000, + "addedFileSizeBytes": 104857600, + "removedFileSizeBytes": 0, + "totalFileSizeBytes": 1178599424, + "totalDurationMs": 5000, + "attempts": 1 + } + ] +} +``` + +--- + +## 5. Authorization + +### 5.1 Required Privileges + +This proposal introduces **new dedicated privileges** for reading observability data, following the principle of **separation of duties**. This ensures that: + +- Read-only audit/monitoring access does not require management permissions +- Monitoring tools can access metrics without requiring data read access +- Fine-grained access control is possible for different operational roles + +| Endpoint | Required Privilege | Scope | New Privilege? | +|----------|-------------------|-------|----------------| +| Query Events | `CATALOG_READ_EVENTS` | Catalog | **Yes** | +| List Scan Metrics | `TABLE_READ_METRICS` | Table | **Yes** | +| List Commit Metrics | `TABLE_READ_METRICS` | Table | **Yes** | + +### 5.2 New Privilege Definitions + +| Privilege | Scope | Description | +|-----------|-------|-------------| +| `CATALOG_READ_EVENTS` | Catalog | Read-only access to catalog events (audit log). Does not grant any management capabilities. | +| `TABLE_READ_METRICS` | Table | Read-only access to table scan and commit metrics. Does not grant access to table data. | + +### 5.3 Rationale: Separation of Duties + +Introducing dedicated read-only privileges enables proper **separation of duties**: + +| Use Case | Required Privilege | Why Not Reuse Existing? | +|----------|-------------------|------------------------| +| Security auditor reviewing catalog changes | `CATALOG_READ_EVENTS` | Should not require `CATALOG_MANAGE_METADATA` (management access) | +| Monitoring tool collecting table metrics | `TABLE_READ_METRICS` | Should not require `TABLE_READ_DATA` (data access) | +| Data analyst with table access | `TABLE_READ_DATA` implies `TABLE_READ_METRICS` | Users who can read data can also see metrics about their queries | +| Catalog admin | `CATALOG_MANAGE_METADATA` implies `CATALOG_READ_EVENTS` | Admins can see all events | + +### 5.4 Privilege Hierarchy + +The new privileges fit into the existing hierarchy as follows: + +``` +CATALOG_MANAGE_METADATA + └── CATALOG_READ_EVENTS (implied) + +TABLE_FULL_METADATA / TABLE_READ_DATA + └── TABLE_READ_METRICS (implied) +``` + +This means: +- Users with `CATALOG_MANAGE_METADATA` automatically have `CATALOG_READ_EVENTS` +- Users with `TABLE_READ_DATA` automatically have `TABLE_READ_METRICS` +- But the reverse is **not** true: `CATALOG_READ_EVENTS` does not grant management access, and `TABLE_READ_METRICS` does not grant data access + +### 5.5 Implementation Notes + +New privileges require: +1. Adding entries to `PolarisPrivilege` enum +2. Updating the privilege hierarchy in the authorizer +3. Adding privilege checks in the new API endpoints + +--- + +## 6. OpenAPI Schema + +> **Note:** The OpenAPI specifications below are embedded in this proposal for review context. Upon approval, these should be extracted into separate files for ease of processing and proper integration: +> - **Events API** → `spec/events-service.yml` (new dedicated service spec with base path `/api/events/v1/`) +> - **Metrics Reports API** → `spec/metrics-reports-service.yml` (new dedicated service spec with base path `/api/metrics-reports/v1/`) + +### 6.1 Events API + +Add the following to a new `spec/events-service.yml`: + +> **Note:** The Events API uses a dedicated `/api/events/v1/` namespace to avoid URI path clashes with the Iceberg REST Catalog events API until that spec is approved. The request/response schemas follow Iceberg Events API patterns for future compatibility. + +```yaml +paths: + /v1/{prefix}: + parameters: + - $ref: '#/components/parameters/prefix' + post: + tags: + - Catalog API + summary: Get events for changes to catalog objects + description: > + Returns a sequence of changes to catalog objects (tables, namespaces, views) + that allows clients to efficiently track metadata modifications without polling + individual resources. Consumers track their progress through a continuation-token, + enabling resumable synchronization after downtime or errors. + + This endpoint primarily supports use cases like catalog federation, workflow + triggering, and basic audit capabilities. + + Consumers should be prepared to handle 410 Gone responses when requested sequences + are outside the server's retention window. Consumers should also de-duplicate + received events based on the event's `event-id`. + operationId: getEvents + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/QueryEventsRequest' + responses: + '200': + description: A sequence of change events to catalog objects + content: + application/json: + schema: + $ref: '#/components/schemas/QueryEventsResponse' + '400': + $ref: '#/components/responses/BadRequestErrorResponse' + '401': + $ref: '#/components/responses/UnauthorizedResponse' + '403': + $ref: '#/components/responses/ForbiddenResponse' + '410': + description: Gone - The requested offset is no longer available + content: + application/json: + schema: + $ref: '#/components/schemas/ErrorModel' + '503': + $ref: '#/components/responses/ServiceUnavailableResponse' + '5XX': + $ref: '#/components/responses/ServerErrorResponse' +``` + +### 6.2 Metrics API (New Metrics Reports Service) + +Add the following to a new `spec/metrics-reports-service.yml` (or extend existing management service): + +> **Note:** The metrics API uses `/api/metrics-reports/v1/` as the base path, separate from the management API. This reflects that metrics reports are read-only access to pre-populated data, not catalog management operations. + +```yaml +paths: + /catalogs/{catalogName}/namespaces/{namespace}/tables/{table}: + parameters: + - $ref: '#/components/parameters/catalogName' + - $ref: '#/components/parameters/namespace' + - $ref: '#/components/parameters/table' + get: + operationId: listTableMetrics + summary: List metrics for a table + description: > + Returns metrics reports for the specified table. The type of metrics + (scan or commit) must be specified via the required metricType parameter. + This unified endpoint supports future extensibility as new metric types + are added. + tags: + - Observability + parameters: + - name: metricType + in: query + required: true + description: Type of metrics to retrieve + schema: + type: string + enum: [scan, commit] + - name: pageToken + in: query + schema: + type: string + - name: pageSize + in: query + schema: + type: integer + minimum: 1 + maximum: 1000 + default: 100 + - name: snapshotId + in: query + schema: + type: integer + format: int64 + - name: principalName + in: query + schema: + type: string + - name: timestampFrom + in: query + schema: + type: integer + format: int64 + - name: timestampTo + in: query + schema: + type: integer + format: int64 + - name: operation + in: query + description: Filter by commit operation (only applicable when metricType=commit) + schema: + type: string + enum: [append, overwrite, delete, replace] + responses: + '200': + description: Paginated list of metrics reports + content: + application/json: + schema: + $ref: '#/components/schemas/ListMetricsResponse' + '400': + description: Bad request (e.g., missing metricType, invalid parameter combination) + '403': + description: Insufficient privileges + '404': + description: Table not found +``` + +### 6.3 Events API Schemas (Iceberg-Compatible) + +Add these schemas to `spec/events-service.yml`: + +```yaml +components: + schemas: + QueryEventsRequest: + type: object + properties: + continuation-token: + type: string + description: > + A continuation token to resume fetching events from a previous request. + If not provided, events are fetched from the beginning of the event log + subject to other filters. + page-size: + type: integer + format: int32 + description: > + The maximum number of events to return in a single response. + Servers may return less results than requested. + after-timestamp-ms: + type: integer + format: int64 + description: > + The timestamp in milliseconds to start consuming events from (inclusive). + operation-types: + type: array + items: + $ref: "#/components/schemas/OperationType" + description: Filter events by operation type. + catalog-objects-by-name: + type: array + items: + $ref: "#/components/schemas/CatalogObjectIdentifier" + description: > + Filter events by catalog objects referenced by name (namespaces, tables, views). + For namespaces, events for all containing objects are returned recursively. + catalog-objects-by-id: + type: array + items: + $ref: "#/components/schemas/CatalogObjectUuid" + description: Filter events by table/view UUIDs. + object-types: + type: array + items: + type: string + enum: [namespace, table, view] + description: Filter events by catalog object type. + custom-filters: + type: object + additionalProperties: true + description: Implementation-specific filter extensions. + + QueryEventsResponse: + type: object + required: + - next-page-token + - highest-processed-timestamp-ms + - events + properties: + next-page-token: + type: string + description: > + An opaque continuation token to fetch the next page of events. + highest-processed-timestamp-ms: + type: integer + format: int64 + description: > + The highest event timestamp processed when generating this response. + events: + type: array + items: + $ref: "#/components/schemas/Event" + + Event: + type: object + required: + - event-id + - request-id + - request-event-count + - timestamp-ms + - operation + properties: + event-id: + type: string + description: Unique ID of this event. Clients should deduplicate based on this ID. + request-id: + type: string + description: > + Opaque ID of the request this event belongs to. Events from the same + request share this ID. + request-event-count: + type: integer + description: > + Total number of events generated by this request. + timestamp-ms: + type: integer + format: int64 + description: Timestamp when this event occurred (epoch milliseconds). + actor: + type: object + additionalProperties: true + description: > + The actor who performed the operation (e.g., user, service account). + Content is implementation-specific. + operation: + type: object + description: The operation that was performed. + discriminator: + propertyName: operation-type + mapping: + create-table: "#/components/schemas/CreateTableOperation" + register-table: "#/components/schemas/RegisterTableOperation" + drop-table: "#/components/schemas/DropTableOperation" + update-table: "#/components/schemas/UpdateTableOperation" + rename-table: "#/components/schemas/RenameTableOperation" + create-view: "#/components/schemas/CreateViewOperation" + drop-view: "#/components/schemas/DropViewOperation" + update-view: "#/components/schemas/UpdateViewOperation" + rename-view: "#/components/schemas/RenameViewOperation" + create-namespace: "#/components/schemas/CreateNamespaceOperation" + update-namespace-properties: "#/components/schemas/UpdateNamespacePropertiesOperation" + drop-namespace: "#/components/schemas/DropNamespaceOperation" + custom: "#/components/schemas/CustomOperation" + + OperationType: + type: string + description: > + Defines the type of operation. Clients should ignore unknown operation types. + anyOf: + - type: string + enum: + - create-table + - register-table + - drop-table + - update-table + - rename-table + - create-view + - drop-view + - update-view + - rename-view + - create-namespace + - update-namespace-properties + - drop-namespace + - $ref: '#/components/schemas/CustomOperationType' + + CustomOperationType: + type: string + description: > + Custom operation type for catalog-specific extensions. + Must start with 'x-' followed by an implementation-specific identifier. + pattern: '^x-[a-zA-Z0-9-_.]+$' + + CustomOperation: + type: object + description: Extension point for catalog-specific operations (e.g., Polaris privileges). + required: + - operation-type + - custom-type + properties: + operation-type: + type: string + const: "custom" + custom-type: + $ref: '#/components/schemas/CustomOperationType' + identifier: + $ref: "#/components/schemas/TableIdentifier" + description: Table or view identifier this operation applies to, if applicable. + namespace: + $ref: "#/components/schemas/Namespace" + description: Namespace this operation applies to, if applicable. + table-uuid: + type: string + format: uuid + view-uuid: + type: string + format: uuid + additionalProperties: true + + CatalogObjectIdentifier: + type: array + items: + type: string + description: Reference to a named object in the catalog (namespace, table, or view). + example: ["accounting", "tax"] + + CatalogObjectUuid: + type: object + required: + - uuid + - type + properties: + uuid: + type: string + description: The UUID of the catalog object. + type: + type: string + enum: [table, view] +``` + +### 6.4 Metrics API Schemas (Polaris Management Service) + +Add these schemas to `spec/polaris-management-service.yml`: + +```yaml +components: + schemas: + ScanMetricsReport: + type: object + required: + - reportId + - catalogId + - tableId + - timestampMs + properties: + reportId: + type: string + catalogId: + type: integer + format: int64 + tableId: + type: integer + format: int64 + timestampMs: + type: integer + format: int64 + principalName: + type: string + requestId: + type: string + otelTraceId: + type: string + description: OpenTelemetry trace ID + otelSpanId: + type: string + description: OpenTelemetry span ID + snapshotId: + type: integer + format: int64 + schemaId: + type: integer + filterExpression: + type: string + projectedFieldIds: + type: string + projectedFieldNames: + type: string + resultDataFiles: + type: integer + format: int64 + resultDeleteFiles: + type: integer + format: int64 + totalFileSizeBytes: + type: integer + format: int64 + totalDataManifests: + type: integer + format: int64 + totalDeleteManifests: + type: integer + format: int64 + scannedDataManifests: + type: integer + format: int64 + scannedDeleteManifests: + type: integer + format: int64 + skippedDataManifests: + type: integer + format: int64 + skippedDeleteManifests: + type: integer + format: int64 + skippedDataFiles: + type: integer + format: int64 + skippedDeleteFiles: + type: integer + format: int64 + totalPlanningDurationMs: + type: integer + format: int64 + equalityDeleteFiles: + type: integer + format: int64 + positionalDeleteFiles: + type: integer + format: int64 + indexedDeleteFiles: + type: integer + format: int64 + totalDeleteFileSizeBytes: + type: integer + format: int64 + + ListMetricsResponse: + description: > + Polymorphic response for metrics queries. The concrete type is determined + by the metricType discriminator field. + oneOf: + - $ref: '#/components/schemas/ListScanMetricsResponse' + - $ref: '#/components/schemas/ListCommitMetricsResponse' + discriminator: + propertyName: metricType + mapping: + scan: '#/components/schemas/ListScanMetricsResponse' + commit: '#/components/schemas/ListCommitMetricsResponse' + + ListScanMetricsResponse: + type: object + required: + - metricType + - reports + properties: + nextPageToken: + type: string + description: Cursor for fetching the next page of results + metricType: + type: string + const: scan + description: Discriminator indicating this response contains scan metrics + reports: + type: array + description: Array of scan metrics reports + items: + $ref: '#/components/schemas/ScanMetricsReport' + + ListCommitMetricsResponse: + type: object + required: + - metricType + - reports + properties: + nextPageToken: + type: string + description: Cursor for fetching the next page of results + metricType: + type: string + const: commit + description: Discriminator indicating this response contains commit metrics + reports: + type: array + description: Array of commit metrics reports + items: + $ref: '#/components/schemas/CommitMetricsReport' + + CommitMetricsReport: + type: object + required: + - reportId + - catalogId + - tableId + - timestampMs + - snapshotId + - operation + properties: + reportId: + type: string + catalogId: + type: integer + format: int64 + tableId: + type: integer + format: int64 + timestampMs: + type: integer + format: int64 + principalName: + type: string + requestId: + type: string + otelTraceId: + type: string + otelSpanId: + type: string + snapshotId: + type: integer + format: int64 + sequenceNumber: + type: integer + format: int64 + operation: + type: string + description: Commit operation (append, overwrite, delete, replace) + addedDataFiles: + type: integer + format: int64 + removedDataFiles: + type: integer + format: int64 + totalDataFiles: + type: integer + format: int64 + addedDeleteFiles: + type: integer + format: int64 + removedDeleteFiles: + type: integer + format: int64 + totalDeleteFiles: + type: integer + format: int64 + addedEqualityDeleteFiles: + type: integer + format: int64 + removedEqualityDeleteFiles: + type: integer + format: int64 + addedPositionalDeleteFiles: + type: integer + format: int64 + removedPositionalDeleteFiles: + type: integer + format: int64 + addedRecords: + type: integer + format: int64 + removedRecords: + type: integer + format: int64 + totalRecords: + type: integer + format: int64 + addedFileSizeBytes: + type: integer + format: int64 + removedFileSizeBytes: + type: integer + format: int64 + totalFileSizeBytes: + type: integer + format: int64 + totalDurationMs: + type: integer + format: int64 + attempts: + type: integer +``` + +--- + +## 7. Implementation Notes + +### 7.1 Prerequisite: Extend Event Persistence Layer + +> **Important:** The current `PolarisPersistenceEventListener` in Apache Polaris only persists **two event types**: `AFTER_CREATE_TABLE` and `AFTER_CREATE_CATALOG`. All other events are ignored. For the Events API to be useful, the persistence layer must be extended to capture all relevant mutation events. + +#### Current State + +The existing event listener (`PolarisPersistenceEventListener.java`) has a limited switch statement: + +```java +public void onEvent(PolarisEvent event) { + switch (event.type()) { + case AFTER_CREATE_TABLE -> handleAfterCreateTable(event); + case AFTER_CREATE_CATALOG -> handleAfterCreateCatalog(event); + default -> { + // Other events not handled by this listener + } + } +} +``` + +#### Required Changes + +The persistence layer needs to be extended to capture all `AFTER_*` mutation events that should be exposed via the Events API: Review Comment: That's where I think we are being too narrow-minded since the beginning about events. **We cannot / should not predict what events are going to be used for**. If you don't want to expose the before events in the REST API, fine: I hear your arguments and I'm OK with it. But other users may have another usage for before events, and may desire to see them persisted in the database and/or showing up in CloudWatch. Other users may decide that they want to persist just the after table events but not the after catalog ones, etc. I would prefer to see the events feature moving towards a world where THE USER decides which events should be passed to the listener(s). Then the listener(s) would just process every event that is received. @nandorKollar is working on something in this direction in https://github.com/apache/polaris/pull/3973. Let's make sure that we eventually make events in Polaris a fully-configurable feature, and not just a feature that satisfies this or that use case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
