Thanks, Arnab, for putting this together, and Dmitri for the suggestions. Would it make sense to offer this through the Polaris CLI? The person changing a catalog’s settings should also be able to check the impact, using the same credentials and workflow. Requiring database access makes that harder, especially when catalog admins and deployment admins are different people.
A management API could support checking one catalog or an entire realm, with the CLI displaying the results. Validation would still run on the server and reuse the existing logic. For a realm check, we should ensure the caller can inspect every catalog so the report is complete. The Admin Tool could still be useful when the server is unavailable, but the CLI seems easier for everyday use. Yufei On Mon, Nov 16, 2026 at 8:27 AM Dmitri Bourlatchkov via dev < [email protected]> wrote: > HI Arnab, > > Good idea about an Admin command for this check! > > Since it works both for the structured and unstructured cases, I'd propose > naming it "check-entity-locations". > > The "--assume-structured" is probably not necessary. The Admin Tool can get > it from its RealmConfig, which is settable independently of the Server > (both as a -D java CLI parameter and .properties entry). The tool should > document this in the "help" message. > > AdminToolProducers will probably need adjustments to produce a real > RealmConfig. > > It might be a bit inconvenient for end users, but it will ensure that the > validation code shared between the Server and Admin Tool uses consistent > sources (RealmConfig) for this value. > > For JSON output, I'd suggest one line per JSON object. This will hopefully > make shell-based processing easier. > > As the next step we could probably enhance the command to also perform > location overlap checks to reconcile with and double-check runtime overlap > validation. > > Cheers, > Dmitri. > > On Fri, Nov 13, 2026 at 2:45 PM Arnab Karmakar <[email protected]> > wrote: > > > Hi all, > > As we're now very close to releasing 1.9.0, I think we can resume > > the discussion about the command naming and detailed output format for a > > read-only admin report of tables and views that are incompatible with > > structured locations (tracked in [5917]). > > > > The documentation for this behavior has already landed in [5922] and I'd > > like to start working on the command, and I need some consensus/pointers > on > > the command name and output format. Here is a concrete proposal. > > > > Command name > > > > polaris-admin-tool.jar check-structured-locations \ > > --realm <realm> [--catalog <catalog>] \ > > [--assume-structured | --as-if-flag-false] \ > > [--format text|json] > > > > > > - --realm: required. The command walks every catalog in the realm > unless > > --catalog narrows it to one. > > - --assume-structured: evaluates the structured-location rules as if > > ALLOW_UNSTRUCTURED_TABLE_LOCATION=false, even while the flag is still > > true. This is the pre-flight check before disabling the flag. > > - Without that option, the command uses the effective flag value and > > still reports affected entities when the flag is already false. > > - The command is read-only. It never changes configuration, catalog > > metadata or stored data. It does not block disabling the flag and does > > not > > repair anything. > > - I'm open to alternatives such as validate-locations or > > report-unstructured-locations. > > > > Output format > > > > The default is human-readable text. It prints the current effective > setting > > per catalog, then one line per violation, then a summary: > > > > Realm: my-realm > > Catalog: prod ALLOW_UNSTRUCTURED_TABLE_LOCATION=true (catalog > > override) evaluating as: structured > > > > TYPE ENTITY LOCATION REASON > > TABLE ns2.orders s3://bucket/prod/ns1/orders outside > > namespace location s3://bucket/prod/ns2 > > VIEW ns2.orders_v s3://bucket/legacy/orders_v outside > > namespace location s3://bucket/prod/ns2 > > > > Summary: catalogs=1 entities_checked=412 incompatible=2 > > > > --format json prints one object per finding, so operators can script > > remediation: > > > > > > > {"realm":"my-realm","catalog":"prod","namespace":["ns2"],"name":"orders","type":"TABLE", > > "location":"s3://bucket/prod/ns1/orders", > > "namespaceLocations":["s3://bucket/prod/ns2"], > > "reason":"OUTSIDE_NAMESPACE_LOCATION"} > > > > Exit codes > > > > - 0: no incompatible entities found. > > - 1: the report completed and found incompatible entities. > > - A different non-zero code for errors. > > > > Implementation > > > > The command will reuse the server's structured-location validation (the > > check introduced in [5856]). That way the report matches what the server > > enforces. The Admin Tool docs will get usage and sample output. > > > > Feedback and guidance are really welcome. Please let me know if I’m > missing > > anything or if there’s anything I should consider. > > > > Thanks > > > > Arnab > > > > > > [5856]: https://github.com/apache/polaris/pull/5856 > > > > [5917]: https://github.com/apache/polaris/issues/5917 > > [5922]: https://github.com/apache/polaris/pull/5922 > > > > On Thu, Oct 29, 2026 at 10:27 PM Yufei Gu <[email protected]> wrote: > > > > > Two issues were filed: > > > - #5916: Document the impact of disabling > > ALLOW_UNSTRUCTURED_TABLE_LOCATION > > > <https://github.com/apache/polaris/issues/5916> — assigned to > milestone > > > *1.9.0*. > > > - #5917: Add an admin report for incompatible table/view locations > > > <https://github.com/apache/polaris/issues/5917> — optional read-only > > > tooling, planned as a follow-up after 1.9.0. > > > Yufei > > > > > > > > > On Thu, Oct 29, 2026 at 12:20 AM Jean-Baptiste Onofré <[email protected] > > > > > wrote: > > > > > > > Hi guys, > > > > > > > > I agree with Alex and Russell: we should not block true -> false. > > > > > > > > Let's start with prep step before the 1.9.0 release as you proposed: > > > > 1. We document the caveat in the configuration docs and upgrade > notes. > > > > 2. The report command to the admin tool makes sense. A read-only > > > > command that walks the catalog (or realm) and lists the tables/views > > > > whose locations would fail the structured check. That can come in a > > > > follow-up after 1.9.0 release. > > > > > > > > Regards > > > > JB > > > > > > > > On Wed, Oct 28, 2026 at 5:40 PM Yufei Gu <[email protected]> > wrote: > > > > > > > > > > I got that a flawed feature isn't a bug :-). My main concern is > that > > it > > > > > breaks other important features, like credential vending and table > > > > updates, > > > > > which occur much more frequently than updating the flag. Are we OK > > with > > > > > this? Maybe it's fine given that only the catalog admin has the > > > > permission. > > > > > Let's document this caveat first. Having a tool to check all > > > > > locations under a catalog is also a good idea. > > > > > > > > > > Yufei > > > > > > > > > > > > > > > On Wed, Oct 28, 2026 at 8:23 AM Russell Spitzer < > > > > [email protected]> > > > > > wrote: > > > > > > > > > > > I beat you to it Alex :) I see we are thinking alike. > > > > > > > > > > > > On Wed, Oct 28, 2026 at 10:21 AM Alex Dutra <[email protected]> > > > wrote: > > > > > > > > > > > > > Hi Yufei, > > > > > > > > > > > > > > I agree that the situation is tricky. > > > > > > > > > > > > > > Option 1 is indeed expensive, but option 2 would actually > prevent > > > > > > > someone who cleaned up their warehouse so that all tables > comply, > > > > from > > > > > > > genuinely switching to structured mode. Conceptually, this > feels > > > > wrong > > > > > > > to me, because we are preventing a transition from a less > secure > > > > setup > > > > > > > to a more secure setup. > > > > > > > > > > > > > > Another concern I have is a practical one: if the flag is set > at > > > the > > > > > > > realm level, how is Polaris going to detect the change? > > > > > > > > > > > > > > Instead, how about a mix of documentation + a dedicated tool > that > > > > > > > could report all the tables that *would* fail if structured > mode > > is > > > > > > > switched on? > > > > > > > > > > > > > > Thanks, > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 28, 2026 at 1:19 AM Yufei Gu <[email protected] > > > > > > wrote: > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > While reviewing #5856 ( > > > https://github.com/apache/polaris/pull/5856), > > > > I > > > > > > > > noticed that we allow changing > > ALLOW_UNSTRUCTURED_TABLE_LOCATION > > > > from > > > > > > > true > > > > > > > > to false without checking existing table/view locations. This > > can > > > > break > > > > > > > > tables that were valid under the previous setting. > > > > > > > > > > > > > > > > For example: > > > > > > > > > > > > > > > > - Create ns1.t1 at its default location with the flag > > enabled. > > > > > > > > - Rename it to ns2.t2. The data stays under ns1. > > > > > > > > - Turn the flag off. Loading the table with credentials > can > > > now > > > > fail > > > > > > > > with a 403 because its location is outside ns2. > > > > > > > > > > > > > > > > The catalog update succeeds, but the problem shows up when > > > someone > > > > > > > accesses > > > > > > > > the table later. > > > > > > > > > > > > > > > > I see two options: > > > > > > > > > > > > > > > > 1. Verify every table/view before allowing the change, and > > > > reject it > > > > > > > if > > > > > > > > any location is incompatible. This could be very expensive > > for > > > > large > > > > > > > > catalogs. We can consider it as phase 2. > > > > > > > > 2. Block true → false altogether for now. This should also > > > cover > > > > > > > > removing the property when that makes the effective value > > > > false. We > > > > > > > can > > > > > > > > still allow false → true. > > > > > > > > > > > > > > > > I’d lean toward option 2 for now. It seems better to reject > the > > > > > > > > configuration change than let it succeed and break existing > > > tables > > > > > > > > afterward. > > > > > > > > > > > > > > > > Any concerns with this approach? > > > > > > > > > > > > > > > > > > > > > > >
