Hi JB, Thanks for updating the PR and putting together this overview. Using an Iceberg table to make these files discoverable sounds useful.
Before adding this as a beta, I think we should clarify how the directory and its inventory table work together. Can users change or drop the inventory table directly? And what happens to the table and the actual files when a directory is deleted? I'm also curious how the sync scan handles large directories, timeouts, and concurrent requests. This depends on the delegation service. An alternative is to create an interface to allow an external scanner, but it has many challenges, like how/when to trigger it and authN/Z handling. The delegation service is still the best way to go. Since access to the files themselves is coming later, we should clearly explain that querying the inventory doesn't automatically give users access to those files. Given that 1.9.0 is close, I'm not sure the delegation service is ready to ship with it. We could have a phased approach though. Yufei On Sun, Oct 18, 2026 at 5:21 AM Jean-Baptiste Onofré <[email protected]> wrote: > > Hi everyone, > > I know that has been discussed for a while, with some periods of pause, but > I want to provide a clear view about the Polaris Directories proposal, and > the status. > To be short: I updated/rebased the PR (#4613), addressing your comments and > implementing your suggestions. > The PR is now ready to review: > > https://github.com/apache/polaris/pull/4613 > > Imho, it would be great to have a beta in Polaris 1.9.0. > > In order to give a clear view for the community, I'm taking a step back > here :) > (I will create a proposal issue with the following content) > > Motivation > -------------- > More and more lakehouse workloads (especially AI) need to work with data > that is not a table, such as images, videos, PDF documents, etc. > These objects often live in the same buckets as Iceberg tables, but they > are invisible to the catalog, so users have to fall back to listing the > object store directly, with no governance and no SQL access. > > Directories make these objects discoverable alongside Iceberg tables in a > Polaris catalog. > > How it works > ------------------ > A directory is a new table-like entity in a namespace. It points to a base > location (prefix) on an object store (S3, GCS, Azure, file) and tracks the > objects it contains in an Iceberg "inventory" table, one row per object, > with the following schema: > > file_uri, content_type, size, checksum_algorithm, checksum, > last_modified, metadata (map<string, string>) > > When a directory is created, Polaris stores its configuration (name, > base_location, include/exclude filters, scan schedule) and creates the > associated inventory table, named <directory>__inventory (a directory and > table can't share a name in a namespace). > > Because the inventory is a regular Iceberg table, any engine that reads > Iceberg can query it, for instance: > > SELECT file_uri, size FROM cat.ns.`images__inventory` > WHERE content_type = 'image/jpeg' ORDER BY last_modified DESC; > > The PR adds: > - New REST endpoints (Polaris specific, next to generic tables and > policies): > GET .../polaris/v1/{prefix}/namespaces/{ns}/directories > POST .../polaris/v1/{prefix}/namespaces/{ns}/directories > GET .../polaris/v1/{prefix}/namespaces/{ns}/directories/{directory} > DELETE .../polaris/v1/{prefix}/namespaces/{ns}/directories/{directory} > POST > .../polaris/v1/{prefix}/namespaces/{ns}/directories/{directory}/scan > The OpenAPI spec is in spec/polaris-catalog-apis/directories-api.yaml > - Location validation: base_location is validated like a table location > (catalog allowed locations, namespace location unless > allow.unstructured.table.location is set), at creation and before each scan. > - A smple, synchronous scan endpoint: it lists the objects under > base_locatiion (with credentials scoped to that location), applies the > filters, and replaces the inventory table content in a single overwrite > commit. A failed scan leaves the table unchanged. It populates file_uri, > content_type (guessed from the file name), size and last_modified. > - A DirectoryScanService SPI, so the scan is pluggable: the build-in > implementation is a @DefaultBean that an extension can replace (e.g. to > compute checksums, read object metadata or update the table incrementally). > - Two feature flags: ENABLE_DIRECTORIES and ENABLE_DIRECTORY_SCAN. > - Documentation: site/content/in-dev/unreleased/directories.md > > Polaris does not schedule scans: scan-schedule is informational, and > periodic scans are left to the caller or to an external service (which can > read the directory configuration through the REST API and write the > inventory table itself). > > Feedback welcome on > ------------------------------ > 1. API shape: endpoint paths, request/response payloads, and the > <directory>__inventory naminig convention. > 2. Directories currently reuse the table privileges (LIST_TABLES, > CREATE_TABLE_DIRECT, LOAD_TABLE, DROP_TABLE_WITHOUT_PURGE, UPDATE_TABLE for > the scan). I don't want to introduce dedicated directory privileges, but > I'm happy to revisit it. > 3. Server-side scan: that's a simple on-demand scan. If users want > something richer we can provide new scan services thanks to the SPI. I > would like to keep it simple and self-provided. > 4. The two feature flags are true by default. I'm happy to change to false > by default if it helps to land Directories in Polaris 1.9.0. > 5. The simple scan service adds iceberg-data, iceberg-parquet and > parquet-hadoop dependencies to the Polaris runtime service (same Parquet > version already on the classpath). > > Follow-ups > --------------- > - Credential vending for the objects themselves via the "indirection URL" > (giving engines access to the files referenced in the inventory) is not > addressed in the PR. I propose to tackle credential vending as a follow-up. > - The scan endpoint doesn't emit Polaris events yet. > - The PR includes unit tests and an end-to-end tests on a file-based > catalog. I did "manual tests" on S3. I haven't tested it on GCS or Azure > yet. > > Thanks to those who already did first rounds of review on the PR. Any > feedback, here or on the PR, is welcome. > > As said earlier, I would love to include Directories as a beta feature in > Polaris 1.9.0. > > Thanks! > > Regards > JB
