Hi everyone, I know that has been discussed for a while, with some periods of pause, but I want to provide a clear view about the Polaris Directories proposal, and the status. To be short: I updated/rebased the PR (#4613), addressing your comments and implementing your suggestions. The PR is now ready to review:
https://github.com/apache/polaris/pull/4613 Imho, it would be great to have a beta in Polaris 1.9.0. In order to give a clear view for the community, I'm taking a step back here :) (I will create a proposal issue with the following content) Motivation -------------- More and more lakehouse workloads (especially AI) need to work with data that is not a table, such as images, videos, PDF documents, etc. These objects often live in the same buckets as Iceberg tables, but they are invisible to the catalog, so users have to fall back to listing the object store directly, with no governance and no SQL access. Directories make these objects discoverable alongside Iceberg tables in a Polaris catalog. How it works ------------------ A directory is a new table-like entity in a namespace. It points to a base location (prefix) on an object store (S3, GCS, Azure, file) and tracks the objects it contains in an Iceberg "inventory" table, one row per object, with the following schema: file_uri, content_type, size, checksum_algorithm, checksum, last_modified, metadata (map<string, string>) When a directory is created, Polaris stores its configuration (name, base_location, include/exclude filters, scan schedule) and creates the associated inventory table, named <directory>__inventory (a directory and table can't share a name in a namespace). Because the inventory is a regular Iceberg table, any engine that reads Iceberg can query it, for instance: SELECT file_uri, size FROM cat.ns.`images__inventory` WHERE content_type = 'image/jpeg' ORDER BY last_modified DESC; The PR adds: - New REST endpoints (Polaris specific, next to generic tables and policies): GET .../polaris/v1/{prefix}/namespaces/{ns}/directories POST .../polaris/v1/{prefix}/namespaces/{ns}/directories GET .../polaris/v1/{prefix}/namespaces/{ns}/directories/{directory} DELETE .../polaris/v1/{prefix}/namespaces/{ns}/directories/{directory} POST .../polaris/v1/{prefix}/namespaces/{ns}/directories/{directory}/scan The OpenAPI spec is in spec/polaris-catalog-apis/directories-api.yaml - Location validation: base_location is validated like a table location (catalog allowed locations, namespace location unless allow.unstructured.table.location is set), at creation and before each scan. - A smple, synchronous scan endpoint: it lists the objects under base_locatiion (with credentials scoped to that location), applies the filters, and replaces the inventory table content in a single overwrite commit. A failed scan leaves the table unchanged. It populates file_uri, content_type (guessed from the file name), size and last_modified. - A DirectoryScanService SPI, so the scan is pluggable: the build-in implementation is a @DefaultBean that an extension can replace (e.g. to compute checksums, read object metadata or update the table incrementally). - Two feature flags: ENABLE_DIRECTORIES and ENABLE_DIRECTORY_SCAN. - Documentation: site/content/in-dev/unreleased/directories.md Polaris does not schedule scans: scan-schedule is informational, and periodic scans are left to the caller or to an external service (which can read the directory configuration through the REST API and write the inventory table itself). Feedback welcome on ------------------------------ 1. API shape: endpoint paths, request/response payloads, and the <directory>__inventory naminig convention. 2. Directories currently reuse the table privileges (LIST_TABLES, CREATE_TABLE_DIRECT, LOAD_TABLE, DROP_TABLE_WITHOUT_PURGE, UPDATE_TABLE for the scan). I don't want to introduce dedicated directory privileges, but I'm happy to revisit it. 3. Server-side scan: that's a simple on-demand scan. If users want something richer we can provide new scan services thanks to the SPI. I would like to keep it simple and self-provided. 4. The two feature flags are true by default. I'm happy to change to false by default if it helps to land Directories in Polaris 1.9.0. 5. The simple scan service adds iceberg-data, iceberg-parquet and parquet-hadoop dependencies to the Polaris runtime service (same Parquet version already on the classpath). Follow-ups --------------- - Credential vending for the objects themselves via the "indirection URL" (giving engines access to the files referenced in the inventory) is not addressed in the PR. I propose to tackle credential vending as a follow-up. - The scan endpoint doesn't emit Polaris events yet. - The PR includes unit tests and an end-to-end tests on a file-based catalog. I did "manual tests" on S3. I haven't tested it on GCS or Azure yet. Thanks to those who already did first rounds of review on the PR. Any feedback, here or on the PR, is welcome. As said earlier, I would love to include Directories as a beta feature in Polaris 1.9.0. Thanks! Regards JB
