Thanks for chiming in, Dmitri. I don’t think we need to tie location
validation to deployment or migration. A catalog admin should be able to
run the check against a live Polaris server before turning off the flag.

Yufei


On Tue, Nov 17, 2026 at 5:57 PM Dmitri Bourlatchkov via dev <
[email protected]> wrote:

> Hi Yufei,
>
> The Python CLI can probably work in this use case too. However, as you
> notes it will require changes to the Management API.
>
> An Admin Tool command is standalone and is not attached to a particular API
> version. It only depends on the MetsStore data.
>
> Since validating locations is a task that needs to run on the backend at
> deployment or migration time, I tend to think that the Admin Tool is a
> better fit... and will probably be easier to implement and decommission
> if/when Polaris evolves beyond the need for the
> current ALLOW_UNSTRUCTURED_TABLE_LOCATION feature flag.
>
> Cheers,
> Dmitri.
>
> On Tue, Nov 17, 2026 at 2:01 PM Yufei Gu <[email protected]> wrote:
>
> > Thanks, Arnab, for putting this together, and Dmitri for the suggestions.
> > Would it make sense to offer this through the Polaris CLI? The person
> > changing a catalog’s settings should also be able to check the impact,
> > using the same credentials and workflow. Requiring database access makes
> > that harder, especially when catalog admins and deployment admins are
> > different people.
> >
> > A management API could support checking one catalog or an entire realm,
> > with the CLI displaying the results. Validation would still run on the
> > server and reuse the existing logic. For a realm check, we should ensure
> > the caller can inspect every catalog so the report is complete.
> >
> > The Admin Tool could still be useful when the server is unavailable, but
> > the CLI seems easier for everyday use.
> >
> > Yufei
> >
> >
> > On Mon, Nov 16, 2026 at 8:27 AM Dmitri Bourlatchkov via dev <
> > [email protected]> wrote:
> >
> > > HI Arnab,
> > >
> > > Good idea about an Admin command for this check!
> > >
> > > Since it works both for the structured and unstructured cases, I'd
> > propose
> > > naming it "check-entity-locations".
> > >
> > > The "--assume-structured" is probably not necessary. The Admin Tool can
> > get
> > > it from its RealmConfig, which is settable independently of the Server
> > > (both as a -D java CLI parameter and .properties entry). The tool
> should
> > > document this in the "help" message.
> > >
> > > AdminToolProducers will probably need adjustments to produce a real
> > > RealmConfig.
> > >
> > > It might be a bit inconvenient for end users, but it will ensure that
> the
> > > validation code shared between the Server and Admin Tool uses
> consistent
> > > sources (RealmConfig) for this value.
> > >
> > > For JSON output, I'd suggest one line per JSON object. This will
> > hopefully
> > > make shell-based processing easier.
> > >
> > > As the next step we could probably enhance the command to also perform
> > > location overlap checks to reconcile with and double-check runtime
> > overlap
> > > validation.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Fri, Nov 13, 2026 at 2:45 PM Arnab Karmakar <[email protected]>
> > > wrote:
> > >
> > > > Hi all,
> > > > As we're now very close to releasing 1.9.0, I think we can resume
> > > > the discussion about the command naming and detailed output format
> for
> > a
> > > > read-only admin report of tables and views that are incompatible with
> > > > structured locations (tracked in [5917]).
> > > >
> > > > The documentation for this behavior has already landed in [5922] and
> > I'd
> > > > like to start working on the command, and I need some
> > consensus/pointers
> > > on
> > > > the command name and output format. Here is a concrete proposal.
> > > >
> > > > Command name
> > > >
> > > > polaris-admin-tool.jar check-structured-locations \
> > > >   --realm <realm> [--catalog <catalog>] \
> > > >   [--assume-structured | --as-if-flag-false] \
> > > >   [--format text|json]
> > > >
> > > >
> > > >    - --realm: required. The command walks every catalog in the realm
> > > unless
> > > >    --catalog narrows it to one.
> > > >    - --assume-structured: evaluates the structured-location rules as
> if
> > > >    ALLOW_UNSTRUCTURED_TABLE_LOCATION=false, even while the flag is
> > still
> > > >    true. This is the pre-flight check before disabling the flag.
> > > >    - Without that option, the command uses the effective flag value
> and
> > > >    still reports affected entities when the flag is already false.
> > > >    - The command is read-only. It never changes configuration,
> catalog
> > > >    metadata or stored data. It does not block disabling the flag and
> > does
> > > > not
> > > >    repair anything.
> > > >    - I'm open to alternatives such as validate-locations or
> > > >    report-unstructured-locations.
> > > >
> > > > Output format
> > > >
> > > > The default is human-readable text. It prints the current effective
> > > setting
> > > > per catalog, then one line per violation, then a summary:
> > > >
> > > > Realm: my-realm
> > > > Catalog: prod   ALLOW_UNSTRUCTURED_TABLE_LOCATION=true (catalog
> > > > override)   evaluating as: structured
> > > >
> > > > TYPE   ENTITY          LOCATION                        REASON
> > > > TABLE  ns2.orders      s3://bucket/prod/ns1/orders     outside
> > > > namespace location s3://bucket/prod/ns2
> > > > VIEW   ns2.orders_v    s3://bucket/legacy/orders_v     outside
> > > > namespace location s3://bucket/prod/ns2
> > > >
> > > > Summary: catalogs=1 entities_checked=412 incompatible=2
> > > >
> > > > --format json prints one object per finding, so operators can script
> > > > remediation:
> > > >
> > > >
> > > >
> > >
> >
> {"realm":"my-realm","catalog":"prod","namespace":["ns2"],"name":"orders","type":"TABLE",
> > > >  "location":"s3://bucket/prod/ns1/orders",
> > > >  "namespaceLocations":["s3://bucket/prod/ns2"],
> > > >  "reason":"OUTSIDE_NAMESPACE_LOCATION"}
> > > >
> > > > Exit codes
> > > >
> > > >    - 0: no incompatible entities found.
> > > >    - 1: the report completed and found incompatible entities.
> > > >    - A different non-zero code for errors.
> > > >
> > > > Implementation
> > > >
> > > > The command will reuse the server's structured-location validation
> (the
> > > > check introduced in [5856]). That way the report matches what the
> > server
> > > > enforces. The Admin Tool docs will get usage and sample output.
> > > >
> > > > Feedback and guidance are really welcome. Please let me know if I’m
> > > missing
> > > > anything or if there’s anything I should consider.
> > > >
> > > > Thanks
> > > >
> > > > Arnab
> > > >
> > > >
> > > > [5856]: https://github.com/apache/polaris/pull/5856
> > > >
> > > > [5917]: https://github.com/apache/polaris/issues/5917
> > > > [5922]: https://github.com/apache/polaris/pull/5922
> > > >
> > > > On Thu, Oct 29, 2026 at 10:27 PM Yufei Gu <[email protected]>
> > wrote:
> > > >
> > > > > Two issues were filed:
> > > > > - #5916: Document the impact of disabling
> > > > ALLOW_UNSTRUCTURED_TABLE_LOCATION
> > > > > <https://github.com/apache/polaris/issues/5916> — assigned to
> > > milestone
> > > > > *1.9.0*.
> > > > > - #5917: Add an admin report for incompatible table/view locations
> > > > > <https://github.com/apache/polaris/issues/5917> — optional
> read-only
> > > > > tooling, planned as a follow-up after 1.9.0.
> > > > > Yufei
> > > > >
> > > > >
> > > > > On Thu, Oct 29, 2026 at 12:20 AM Jean-Baptiste Onofré <
> > [email protected]
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi guys,
> > > > > >
> > > > > > I agree with Alex and Russell: we should not block true -> false.
> > > > > >
> > > > > > Let's start with prep step before the 1.9.0 release as you
> > proposed:
> > > > > > 1. We document the caveat in the configuration docs and upgrade
> > > notes.
> > > > > > 2. The report command to the admin tool makes sense. A read-only
> > > > > > command that walks the catalog (or realm) and lists the
> > tables/views
> > > > > > whose locations would fail the structured check. That can come
> in a
> > > > > > follow-up after 1.9.0 release.
> > > > > >
> > > > > > Regards
> > > > > > JB
> > > > > >
> > > > > > On Wed, Oct 28, 2026 at 5:40 PM Yufei Gu <[email protected]>
> > > wrote:
> > > > > > >
> > > > > > > I got that a flawed feature isn't a bug :-). My main concern is
> > > that
> > > > it
> > > > > > > breaks other important features, like credential vending and
> > table
> > > > > > updates,
> > > > > > > which occur much more frequently than updating the flag. Are we
> > OK
> > > > with
> > > > > > > this? Maybe it's fine given that only the catalog admin has the
> > > > > > permission.
> > > > > > > Let's document this caveat first. Having a tool to check all
> > > > > > > locations under a catalog is also a good idea.
> > > > > > >
> > > > > > > Yufei
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Oct 28, 2026 at 8:23 AM Russell Spitzer <
> > > > > > [email protected]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I beat you to it Alex :) I see we are thinking alike.
> > > > > > > >
> > > > > > > > On Wed, Oct 28, 2026 at 10:21 AM Alex Dutra <
> [email protected]
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Yufei,
> > > > > > > > >
> > > > > > > > > I agree that the situation is tricky.
> > > > > > > > >
> > > > > > > > > Option 1 is indeed expensive, but option 2 would actually
> > > prevent
> > > > > > > > > someone who cleaned up their warehouse so that all tables
> > > comply,
> > > > > > from
> > > > > > > > > genuinely switching to structured mode. Conceptually, this
> > > feels
> > > > > > wrong
> > > > > > > > > to me, because we are preventing a transition from a less
> > > secure
> > > > > > setup
> > > > > > > > > to a more secure setup.
> > > > > > > > >
> > > > > > > > > Another concern I have is a practical one: if the flag is
> set
> > > at
> > > > > the
> > > > > > > > > realm level, how is Polaris going to detect the change?
> > > > > > > > >
> > > > > > > > > Instead, how about a mix of documentation + a dedicated
> tool
> > > that
> > > > > > > > > could report all the tables that *would* fail if structured
> > > mode
> > > > is
> > > > > > > > > switched on?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Alex
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Oct 28, 2026 at 1:19 AM Yufei Gu <
> > [email protected]
> > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > While reviewing #5856 (
> > > > > https://github.com/apache/polaris/pull/5856),
> > > > > > I
> > > > > > > > > > noticed that we allow changing
> > > > ALLOW_UNSTRUCTURED_TABLE_LOCATION
> > > > > > from
> > > > > > > > > true
> > > > > > > > > > to false without checking existing table/view locations.
> > This
> > > > can
> > > > > > break
> > > > > > > > > > tables that were valid under the previous setting.
> > > > > > > > > >
> > > > > > > > > > For example:
> > > > > > > > > >
> > > > > > > > > >    - Create ns1.t1 at its default location with the flag
> > > > enabled.
> > > > > > > > > >    - Rename it to ns2.t2. The data stays under ns1.
> > > > > > > > > >    - Turn the flag off. Loading the table with
> credentials
> > > can
> > > > > now
> > > > > > fail
> > > > > > > > > >    with a 403 because its location is outside ns2.
> > > > > > > > > >
> > > > > > > > > > The catalog update succeeds, but the problem shows up
> when
> > > > > someone
> > > > > > > > > accesses
> > > > > > > > > > the table later.
> > > > > > > > > >
> > > > > > > > > > I see two options:
> > > > > > > > > >
> > > > > > > > > >    1. Verify every table/view before allowing the change,
> > and
> > > > > > reject it
> > > > > > > > > if
> > > > > > > > > >    any location is incompatible. This could be very
> > expensive
> > > > for
> > > > > > large
> > > > > > > > > >    catalogs. We can consider it as phase 2.
> > > > > > > > > >    2. Block true → false altogether for now. This should
> > also
> > > > > cover
> > > > > > > > > >    removing the property when that makes the effective
> > value
> > > > > > false. We
> > > > > > > > > can
> > > > > > > > > >    still allow false → true.
> > > > > > > > > >
> > > > > > > > > > I’d lean toward option 2 for now. It seems better to
> reject
> > > the
> > > > > > > > > > configuration change than let it succeed and break
> existing
> > > > > tables
> > > > > > > > > > afterward.
> > > > > > > > > >
> > > > > > > > > > Any concerns with this approach?
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Reply via email to