Hi guys,
Thanks for your feedback. Let me describe what the PR already does for each
of your questions:
> Can users change or drop the inventory table directly? And what happens
> to the table and the actual files when a directory is deleted?
The directory and its inventory are two separate entities in the namespace:
- The directory is a new table-like subtype (DIRECTORY). It can only be
managed through the directories endpoints: the Iceberg API doesn't see it
as a table (dropTable on a directory returns false and leaves it intact).
- The inventory (<directory>__inventory) is a regular Iceberg table,
created by Polaris together with the directory. If a table with that name
already exists, the directory creation is rejected. If the table creation
fails, the directory creation is rolled back.
- Dropping a directory drops the inventory table first (without purge),
then the directory entity. If the table drop fails, the directory is kept
and the drop can be retried.
- Polaris never deletes the object under base_location: they are not
managed by the catalog, and dropping a directory doesn't toucn them.
Today the inventory is not protected beyond the regular table privileges
(as I mentioned in my previous email): a user with write access can change
it, and the next scan overwrites the changes (last scan wins). A user with
drop privilege can drop it, and the scan then fails until the directory is
recreated. I can document this clearly, and I can protect the inventory
against direct drop/write if we think it's needed for the beta.
> I'm also curious how the sync scan handles large directories, timeouts,
> and concurrent requests.
Remember that initially, Directories proposal didn't include any scan (it
was a delegation service).
During the discussion, several people expressed that Directories without a
simple scan by default would be confusing and useless (I think it was
Romain).
That's why I added a simple scan service.
What the scan does in the PR:
- It's an explicit, on-demand call (POST .../directories/{directory}/scan).
Polaris doesn't schedule anything, scan-schedule is informational
(scheduling can be done externally with cron, Airflow, etc).
- It lists base_location with read/list credentials scoped to that
location, and validates base_location against the allowed locations again
before each scan (in case the storage config changed after the directory
creation).
- It replaces the inventory content in a single Iceberg overwrite commit,
so a failed scan leaves the table unchanged (I fixed the
CommitStateUnknownException case Ankur found).
- About concurrent scans, the overwrite uses an "always true" row filter,
which Iceberg re-applies when it retries a conflicting commit. Two
concurrent scans don't duplicate rows, the last commit wins.
- About large directories, the listing is kept in memory and the scan runs
inside the request, so it's not designed for millions of objects. That's
why the scan has its own feature flag (ENABLE_DIRIECTORY_SCAN), separate
from ENABLE_DIRECTORIES. I'm completely fine to set if to false by default
for 1.9.0 (it was my proposal in my previous email).
I think I was clear that "advanced" scan services should be a delegation
service, where integrators and vendors can implement the way they want.
To do that, the PR already includes:
- The scan is behinid a DirectoryScanService SPI. The built-in
implementation is a @DefaultBean, and an extension can replace it.
- An external scanner doesn't need a new interface: it reads the directory
configuration with the directories REST API, and writes the inventory as a
regular Iceberg table, with the regular table privileges. It uses its own
credentials to list the object store, and decides when to run.
> Since access to the files themselves is coming later, we should clearly
> explain that querying the inventory doesn't automatically give users
> access to those files.
I agree, and that's exactly the current behavior: the PR doesn't vend any
credentials for the objects listed in the inventory. Loading the inventory
only gives access to its own data and metadata files, like any Iceberg
table. The documentation is probably not explicit enough (the "Credential
vending" section is probably too vague), I will update it.
Credential vending for the objects (the "indirection URL") is a follow-up.
> Given that 1.9.0 is close, I'm not sure the delegation service is ready
> to ship with it. We could have a phased approach though.
That works for me, and I'm a big fan of phased approach :)
That's why I scoped the PR to the first phase:
1. 1.9.0 (beta): directories endpoints (create, list, load, drop, ...), the
inventory table, the scan SPI, and the simple built-in scan behind
ENABLE_DIRECTORY_SCAN (false by default).
2. Scan moved to the delegation service (or an async task) when ready, with
scan events.
3. Credential vending for the objects.
The PR has unit tests and an end-to-end test on a file-based catalog
(create, scan, load, drop, and creation outside the allowed locations
rejected). As said, I tested manually on S3.
About Tornike's feedback:
- Today the bucket is the source of truth, and the inventory is an index
refreshed by scans. That's deliberate, so existing objects written by tools
that don't know Polaris can be made discoverable (that's the main
motivation).
- Each scan commiits in one go, so readers never see a partial listing.
- The snapshot commit time shows when the listing was last refreshed.
- Time travel gives earlier listings.
- Between scans the inventory can miss objects or list deleted ones. And I
think it's totally fine.
- In the index model, direct writes to the inventory get overwritten by the
next scan. In this model, they would be how objects get written.
- The PR doesn't "block" this model:
- The directory and the inventory are separate, with a fixed schema.
- The scan is optional and pluggable.
- Limiting credential vending to paths listed in the inventory can work
under either model
I think it's the most flexible model, and it's the one I propose to 1.9.0.
I will work on a follow-up model discussion (after 1.9.0), more a "managed"
mode.
Thanks!
Regards
JB
On Tue, Oct 20, 2026 at 6:22 AM Tornike Gurgenidze <[email protected]>
wrote:
> Hi JB,
>
> Thanks for the write-up. +1 on the motivation.
>
> I might be going too off-course here, but I'd like to share a slightly
> different angle on what a Directory could become.
>
> Before looking at the PR, my mental model of a Polaris Directory was a
> metadata layer over a file system, the same way Iceberg is a metadata layer
> over Parquet files: the inventory table is the source of truth, not the
> bucket. Like Iceberg manifests, an object that isn't in the inventory
> doesn't exist, and deleting the inventory entry is what deletes the file;
> the physical object is just garbage to be cleaned up later.
>
> With that convention, it becomes possible to implement a real file system
> on top of Polaris, e.g. a Hadoop FileSystem or fsspec:
> polaris://catalog/ns/images/2004/a.jpg resolved through the directory, with
> access governed by Polaris (vended credentials or remote signing, only for
> paths that exist in the inventory). We'd also get consistent,
> snapshot-based listings, atomic multi-file commits, time travel, and
> natural support for the per-object credential vending discussed as a
> follow-up.
>
> Unless I'm misreading the direction of the proposal, the current PR goes
> the other way: the bucket is the source of truth and the scan replaces the
> inventory with whatever is listed. My concern with this is that it splits
> the source of truth between writers and readers. Writers write directly to
> the object store, so for them the bucket is the truth. Anyone accessing the
> data through Polaris necessarily treats the inventory as the truth. The two
> can only be reconciled by a scan, so between scans there's always a window
> where the inventory is stale: new objects are missing, deleted objects are
> still listed, and changed objects have outdated metadata. Readers going
> through Polaris can't tell when that's the case.
>
> That's why treating Polaris as the source of truth for both writers and
> readers makes more sense to me. If writes go through Polaris as well, there
> is only one view of what the directory contains, and readers and writers
> always agree on it, the same guarantee Iceberg gives for tables.
>
> Curious what you and others think.
>
> Thanks,
> Tornike
>
>
> On Tue, Oct 20, 2026 at 1:48 AM Ankur Kotwal <[email protected]>
> wrote:
>
> > Hi JB,
> >
> > Thanks for the clear write-up. I went through the PR, mostly the scan
> path
> > (DirectoryScanner, DirectoryCatalogHandler.scanDirectory) since that is
> > where I have been spending time with commits. +1 on the overall
> direction.
> > Some notes, the first one is the one I'd fix before 1.9.0.
> >
> > 1/ Cleanup on CommitStateUnknownException
> >
> > In DirectoryScanner.scan, any RuntimeException from overwrite.commit()
> > deletes the new Parquet file. If the commit fails with
> > CommitStateUnknownException the overwrite may actually have landed, and
> > then the current snapshot points at a file we just deleted, so every read
> > of the inventory fails. Iceberg's own SnapshotProducer skips cleanup for
> > that exception for this reason. Suggest catching
> > CommitStateUnknownException first and rethrowing it without deleting,
> and a
> > unit test with a table ops that throws it after committing.
> >
> > 2/ Memory and request time
> >
> > The scan collects every record in a List before writing, inside the HTTP
> > request. A prefix with a few million objects means a few million Records
> on
> > the heap, and a request that can easily outlive a proxy or client
> timeout.
> > Two cheap options: stream the listing into a rolling writer (one file
> per N
> > rows) instead of a list, and add a configurable max object count that
> fails
> > the scan with a clear 4xx when exceeded. Longer term this is a good fit
> for
> > an async task, but that can wait.
> >
> > 3/ Base location without a trailing slash
> >
> > The listing uses the base location as the raw prefix. With base_location
> =
> > s3://bucket/images, the ListObjectsV2 prefix is "images", which also
> > matches images-old/ and images2/. And I think the vended credentials will
> > refuse it: the list policy condition is s3:prefix "images/*", so on real
> S3
> > the call may get AccessDenied. Normalizing the base location to end with
> > "/" at creation (or before listing) would fix both. Worth a quick check
> > since the manual S3 test may have used a trailing slash.
> >
> > 4/ Inventory table inside the scanned prefix
> >
> > The inventory table is created with the default location, under the
> > namespace location. If the base location covers the namespace location
> (for
> > example a directory over the whole namespace prefix), each scan records
> the
> > inventory's own data and metadata files, and the inventory grows with
> every
> > scan. Either reject that overlap at creation or exclude the inventory
> table
> > location from the listing.
> >
> > 5/ Filters
> >
> > include/exclude are Java regexes compiled at scan time. Two small things:
> > compile them in createDirectory so a bad pattern is a 400 at creation
> > instead of a failed scan later, and since any principal who can create
> > tables in the namespace can create a directory, a pattern with
> catastrophic
> > backtracking runs on the server against every object URI. Globs (or a
> > length limit plus a match timeout) would avoid that. Happy either way,
> but
> > it is worth a line in the docs.
> >
> > On your questions:
> >
> > 1. API shape: looks good to me. On <directory>__inventory: since the
> > directory and its inventory are separate entities, I'd return the
> inventory
> > table identifier in LoadDirectoryResponse so clients don't have to
> rebuild
> > the convention, and it leaves room to change the naming later.
> >
> > 2. Privileges: fine to reuse the table privileges for the beta. One gap:
> > the inventory is an ordinary table, so anyone with TABLE_WRITE_DATA on it
> > can rewrite it directly, and the next scan silently overwrites their
> > changes (the overwrite uses alwaysTrue with no conflict validation). That
> > is fine as "last scan wins", but it should be written down, together with
> > the fact that an SPI implementation that adds checksums incrementally
> will
> > race with the built-in full scan.
> >
> > 3. Server-side scan: agree on keeping it simple with the SPI. Points 1
> and
> > 2 above are what I'd want for the simple version.
> >
> > 4. Feature flags: I'd keep ENABLE_DIRECTORIES true and set
> > ENABLE_DIRECTORY_SCAN to false by default for 1.9.0, mainly because of
> the
> > memory/timeout point. Users who opt in know they are running a
> synchronous
> > scan.
> >
> > 5. Dependencies: no concern, same Parquet version.
> >
> > Smaller things for the PR itself: dropDirectory calls dropTable(tableId,
> > false), so the inventory's data files stay in storage after a drop;
> > probably fine but worth documenting. And createDirectory writes the
> entity
> > before the table, so a crash in between leaves a directory with no
> > inventory; scanDirectory then fails on loadTable with a
> > NoSuchTableException, which a clearer message would help with.
> >
> > Thanks again for pushing this, happy to help with tests on the scan path.
> >
> > Regards,
> > Ankur
> >
> > On Sun, Oct 18, 2026 05:21 AM, Jean-Baptiste Onofré <[email protected]>
> > wrote:
> >
> > > Hi everyone,
> > >
> > > I know that has been discussed for a while, with some periods of pause,
> > but
> > > I want to provide a clear view about the Polaris Directories proposal,
> > and
> > > the status.
> > > To be short: I updated/rebased the PR (#4613), addressing your comments
> > and
> > > implementing your suggestions.
> > > The PR is now ready to review:
> > >
> > > https://github.com/apache/polaris/pull/4613
> > >
> > > Imho, it would be great to have a beta in Polaris 1.9.0.
> > >
> > > In order to give a clear view for the community, I'm taking a step back
> > > here :)
> > > (I will create a proposal issue with the following content)
> > >
> > > Motivation
> > > --------------
> > > More and more lakehouse workloads (especially AI) need to work with
> data
> > > that is not a table, such as images, videos, PDF documents, etc.
> > > These objects often live in the same buckets as Iceberg tables, but
> they
> > > are invisible to the catalog, so users have to fall back to listing the
> > > object store directly, with no governance and no SQL access.
> > >
> > > Directories make these objects discoverable alongside Iceberg tables
> in a
> > > Polaris catalog.
> > >
> > > How it works
> > > ------------------
> > > A directory is a new table-like entity in a namespace. It points to a
> > base
> > > location (prefix) on an object store (S3, GCS, Azure, file) and tracks
> > the
> > > objects it contains in an Iceberg "inventory" table, one row per
> object,
> > > with the following schema:
> > >
> > > file_uri, content_type, size, checksum_algorithm, checksum,
> > > last_modified, metadata (map<string, string>)
> > >
> > > When a directory is created, Polaris stores its configuration (name,
> > > base_location, include/exclude filters, scan schedule) and creates the
> > > associated inventory table, named <directory>__inventory (a directory
> and
> > > table can't share a name in a namespace).
> > >
> > > Because the inventory is a regular Iceberg table, any engine that reads
> > > Iceberg can query it, for instance:
> > >
> > > SELECT file_uri, size FROM cat.ns.`images__inventory`
> > > WHERE content_type = 'image/jpeg' ORDER BY last_modified DESC;
> > >
> > > The PR adds:
> > > - New REST endpoints (Polaris specific, next to generic tables and
> > > policies):
> > > GET .../polaris/v1/{prefix}/namespaces/{ns}/directories
> > > POST .../polaris/v1/{prefix}/namespaces/{ns}/directories
> > > GET
> > .../polaris/v1/{prefix}/namespaces/{ns}/directories/{directory}
> > > DELETE
> > .../polaris/v1/{prefix}/namespaces/{ns}/directories/{directory}
> > > POST
> > > .../polaris/v1/{prefix}/namespaces/{ns}/directories/{directory}/scan
> > > The OpenAPI spec is in
> spec/polaris-catalog-apis/directories-api.yaml
> > > - Location validation: base_location is validated like a table location
> > > (catalog allowed locations, namespace location unless
> > > allow.unstructured.table.location is set), at creation and before each
> > > scan.
> > > - A smple, synchronous scan endpoint: it lists the objects under
> > > base_locatiion (with credentials scoped to that location), applies the
> > > filters, and replaces the inventory table content in a single overwrite
> > > commit. A failed scan leaves the table unchanged. It populates
> file_uri,
> > > content_type (guessed from the file name), size and last_modified.
> > > - A DirectoryScanService SPI, so the scan is pluggable: the build-in
> > > implementation is a @DefaultBean that an extension can replace (e.g. to
> > > compute checksums, read object metadata or update the table
> > incrementally).
> > > - Two feature flags: ENABLE_DIRECTORIES and ENABLE_DIRECTORY_SCAN.
> > > - Documentation: site/content/in-dev/unreleased/directories.md
> > >
> > > Polaris does not schedule scans: scan-schedule is informational, and
> > > periodic scans are left to the caller or to an external service (which
> > can
> > > read the directory configuration through the REST API and write the
> > > inventory table itself).
> > >
> > > Feedback welcome on
> > > ------------------------------
> > > 1. API shape: endpoint paths, request/response payloads, and the
> > > <directory>__inventory naminig convention.
> > > 2. Directories currently reuse the table privileges (LIST_TABLES,
> > > CREATE_TABLE_DIRECT, LOAD_TABLE, DROP_TABLE_WITHOUT_PURGE, UPDATE_TABLE
> > for
> > > the scan). I don't want to introduce dedicated directory privileges,
> but
> > > I'm happy to revisit it.
> > > 3. Server-side scan: that's a simple on-demand scan. If users want
> > > something richer we can provide new scan services thanks to the SPI. I
> > > would like to keep it simple and self-provided.
> > > 4. The two feature flags are true by default. I'm happy to change to
> > false
> > > by default if it helps to land Directories in Polaris 1.9.0.
> > > 5. The simple scan service adds iceberg-data, iceberg-parquet and
> > > parquet-hadoop dependencies to the Polaris runtime service (same
> Parquet
> > > version already on the classpath).
> > >
> > > Follow-ups
> > > ---------------
> > > - Credential vending for the objects themselves via the "indirection
> URL"
> > > (giving engines access to the files referenced in the inventory) is not
> > > addressed in the PR. I propose to tackle credential vending as a
> > follow-up.
> > > - The scan endpoint doesn't emit Polaris events yet.
> > > - The PR includes unit tests and an end-to-end tests on a file-based
> > > catalog. I did "manual tests" on S3. I haven't tested it on GCS or
> Azure
> > > yet.
> > >
> > > Thanks to those who already did first rounds of review on the PR. Any
> > > feedback, here or on the PR, is welcome.
> > >
> > > As said earlier, I would love to include Directories as a beta feature
> in
> > > Polaris 1.9.0.
> > >
> > > Thanks!
> > >
> > > Regards
> > > JB
> > >
> >
>
