Hi Serhii,

+1 on keeping the view exception, and the upgrade note is
clear now.

One setup where the overlap consequence you documented
doesn't hold, though. With OPTIMIZED_SIBLING_CHECK on and
ALLOW_UNSTRUCTURED_TABLE_LOCATION off, the validator checks
base locations through the index and then returns without
listing sibling write paths [1].

That shortcut assumes write paths sit under the base
location, which a grandfathered view doesn't guarantee. So
in that mode a new table can be created at a legacy view's
metadata path, and both end up with credentials for the
same prefix. The "first writer wins" behavior Ayush
described only applies to the list-based check.

It needs a non-default flag plus a legacy layout, so I
don't think it should block 6036. I left a note on the PR [2]
asking if the new storage-layout section can call it out.

Thanks,
*Vignesh*

[1]
https://github.com/apache/polaris/blob/dc8eb486d0c983067369cae2fabf4c38a2af1f59/runtime/service/src/main/java/org/apache/polaris/service/catalog/common/LocationOverlapValidator.java#L113-L118
[2]
https://github.com/apache/polaris/pull/6036

On Sat, 14 Nov 2026 at 00:08, Serhii Svynarchuk <[email protected]> wrote:

> Hi Ayush,
>
> Thanks for confirming.
>
> I’ve updated the upgrade notes to explain the deliberate table/view
> difference and the storage-layout page to describe the overlap consequence.
> The changes are pushed to https://github.com/apache/polaris/pull/6036
>
> Thanks,
> Serhii
>
> On Fri, Nov 13, 2026 at 6:27 PM Ayush Saxena <[email protected]> wrote:
>
> > Hi Serhii,
> >
> > Yes, that sounds reasonable. An explicit compatibility exception for
> > views is what I was after — my concern was the two diverging silently,
> > not the direction you have chosen.
> >
> > One thing to fold into the documentation alongside the overlap
> > consequence: that tables and views deliberately differ here, so an
> > operator reading the upgrade notes can see that a legacy table's next
> > commit is fully validated while a legacy view's is not. Stating the
> > difference is most of the value of making it a decision.
> >
> > Agreed on keeping the logging separate, and the documentation covering
> > both uses of the flag is enough for me.
> >
> > Thanks,
> >
> > -Ayush
> >
> > On Fri, 13 Nov 2026 at 21:53, Serhii Svynarchuk <[email protected]>
> > wrote:
> > >
> > > Thanks, Alex, for the review and approval.
> > >
> > > Ayush, yes, the compatibility behavior is deliberate. The aim is to
> avoid
> > > breaking ordinary updates to existing views after an upgrade.
> > > My proposal is to keep that behavior for views in this PR and leave the
> > > existing table behavior unchanged. That would be an explicit
> > compatibility
> > > exception for views.
> > > You’re also right that this is ongoing grandfathering, rather than a
> > > one-time migration exemption. As long as both paths remain unchanged,
> > > commits enforce allowed locations but skip structured-location and
> > overlap
> > > checks. Changing either path triggers full validation.
> > > Overlap consequences can be documented: once a view’s paths are
> recorded,
> > > they can block later creates at overlapping locations, while the
> existing
> > > layout remains accepted.
> > > Logging would be useful for auditing, but I’d prefer to consider that
> > > separately from this fix.
> > > Regarding the flag, the name is imperfect, but the documentation now
> > > explicitly covers its use for both view base locations and view
> metadata
> > > paths.
> > >
> > > Does keeping this compatibility exception for views, with those
> > > consequences documented, sound reasonable?
> > >
> > > Thanks,
> > > Serhii
> > >
> > > On Fri, Nov 13, 2026 at 6:06 PM Ayush Saxena <[email protected]>
> wrote:
> > >
> > > > Hi Serhii,
> > > >
> > > > The credential scoping part is clearly a fix — omitting
> > > > write.metadata.path from the scope while writing to it can only fail,
> > > > and FILE storage hiding it explains why it went unnoticed. No
> > > > objection there.
> > > >
> > > > On the compatibility behaviour you asked about: it is more lenient
> > > > than what tables do today, and I would like to understand whether
> that
> > > > is deliberate. A table last committed before 1.3.0 has all its
> > > > locations validated on its next commit, and that commit fails if they
> > > > do not satisfy the current rules. The proposal for views is that a
> > > > commit leaving the base location and metadata path unchanged keeps
> the
> > > > existing layout, enforcing allowed locations but skipping the
> > > > structured-location and overlap checks. So after this change the same
> > > > legacy situation gives a hard failure on a table and a pass on a
> view.
> > > >
> > > > I can see the argument for either, and not breaking existing views on
> > > > upgrade is a good reason. What I would rather avoid is the two
> > > > diverging without that being an explicit decision, since an operator
> > > > upgrading cannot then reason about existing objects without knowing
> > > > which kind each one is. Could we settle it on this thread either way
> —
> > > > keep the view grandfathering as intended, or plan to align tables
> with
> > > > it later — so that it is recorded as a choice rather than read as an
> > > > inconsistency by whoever next touches this?
> > > >
> > > > A related question: the exemption appears to be keyed on the paths
> > > > staying unchanged rather than on a single migration commit, so a view
> > > > that never changes its paths is never fully checked. That is
> > > > defensible as grandfathering, but it does mean a legacy view can keep
> > > > a layout that a new view could not be created with.
> > > >
> > > > On overlap specifically, recording the locations means other objects
> > > > can see them from then on. So a legacy view whose location overlaps
> > > > something keeps that overlap, while a table or view created
> afterwards
> > > > in the same place is rejected because of it. First writer wins, and
> > > > the one that gets rejected is the one nobody has touched yet. I think
> > > > that is unavoidable if we grandfather at all. The storage-layout page
> > > > already describes the exemption itself, but not this consequence of
> > > > it, and it is worth adding there: the rejection will not look related
> > > > to the view.
> > > >
> > > > Optional: logging when a non-conforming legacy layout is recorded
> > > > would make the grandfathered set auditable, so an operator can find
> > > > out what is exempt without inferring it from failures later.
> > > >
> > > > Minor: ALLOW_UNSTRUCTURED_TABLE_LOCATION already covered view base
> > > > locations, and this change also uses it for view write.metadata.path.
> > > > Reusing it avoids a second flag, but the name still does not say what
> > > > it does.
> > > >
> > > > Thanks for picking this up.
> > > >
> > > > -Ayush
> > > >
> > > > On Fri, 13 Nov 2026 at 21:14, Serhii Svynarchuk <[email protected]>
> > > > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > I’d like to discuss https://github.com/apache/polaris/pull/6036,
> > which
> > > > > addresses custom metadata paths for views and their location
> > validation.
> > > > >
> > > > > Creating or updating a view on S3 could fail with HTTP 403 when
> using
> > > > > scoped credentials and write.metadata.path pointed outside the view
> > base
> > > > > location. Polaris omitted the custom metadata path when scoping the
> > > > > temporary credentials used to write view metadata, so S3 rejected
> the
> > > > write
> > > > > because those credentials didn’t allow access to it. FILE storage
> > didn’t
> > > > > expose this problem because it doesn’t scope credentials.
> > > > >
> > > > > The PR includes the metadata path in the credential scope and
> > applies the
> > > > > same location rules used for tables. By default, a new or changed
> > > > metadata
> > > > > path must be inside the view location. With
> > > > > ALLOW_UNSTRUCTURED_TABLE_LOCATION enabled, it can be elsewhere
> > within the
> > > > > catalog allowed locations, subject to the configured overlap
> checks.
> > > > >
> > > > > It also records view locations for sibling overlap checks and
> > metadata
> > > > > purge, and validates view locations during renames. Purge uses the
> > > > recorded
> > > > > locations after checking that the metadata agrees with them. Older
> > views
> > > > > without recorded locations keep a metadata-based fallback so they
> can
> > > > still
> > > > > be dropped.
> > > > >
> > > > > One point I’d particularly like feedback on is compatibility.
> > Following
> > > > > earlier review feedback, updates that leave both the base location
> > and
> > > > > metadata path unchanged preserve the existing layout, while still
> > > > enforcing
> > > > > the catalog allowed locations. This avoids breaking existing views
> > after
> > > > an
> > > > > upgrade. New views and changed paths get the full validation.
> > > > >
> > > > > For context, Iceberg also removed the deprecation of this property:
> > > > > https://github.com/apache/iceberg/pull/15250
> > > > >
> > > > >
> > > > > Do you have any objections to this approach, particularly the
> > > > compatibility
> > > > > behavior for existing views?
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Serhii
> > > >
> >
>

Reply via email to