Hi JB,

Thanks for the review. I have incorporated the two documentation points.

Two-step migration: the table is a pure addition that no earlier
version reads, so the docs now create it while the previous version is
still serving and record the version last, leaving the restart as the
only window where no node can serve.

Realms purged before the upgrade: the migration section now has an
optional one-off statement that queues every realm with time-series
rows but no entities, and states what skipping it leaves behind. It
needs to run while no realm is being bootstrapped, since a realm is
indistinguishable from a purged one until its first entity is written.

Single v7: sure. This PR introduces v7 today, and I will keep an eye
on #6013 and #4756 — if either gets merged earlier I will update this
one to add to the v7 already there. One thing to flag for that
section: #4756 does not bump the version at all right now, and CREATE
INDEX IF NOT EXISTS keeps the existing four-column index, so its new
definition needs a DROP before the CREATE.

Naming and shared delete logic: agreed, before both merge. The loop
here is already generic over the table and its key column and takes a
cutoff, so an age-based command needs only a different cutoff, not new
deletion logic. Happy to extract it, or to adopt whatever merges
first. On naming, grouping the three under one backend-specific
parent, as the NoSQL commands are, reads better to me than three
top-level commands that work on one backend only.

-Ayush

On Thu, 12 Nov 2026 at 12:08, Jean-Baptiste Onofré <[email protected]> wrote:
>
> Hi Ayush,
>
> I agree that a required migration should not be a surprise in the release
> note (it something we discussed while ago when we "moved" from EclipseLink
> to JDBC afair).
>
> I'm +1 with the approach.
>
> I have a few comments though:
> - We have two PRs touching the schema. #6013 bumps the schema to v7, to add
> an index on events(realm_id, timestamp_ms), while #4756 changes the
> definition of idx_scan_report_lookup and idx_commit_report_lookup. As
> CREATE INDEX IF NOT EXISTS keeps the existing index, existing deployment
> won't get the new definition without a migration (same situation as
> idx_locations in v6). As 1.9.0 is already on vote, it means these changes
> will land in 1.10.0. I would prefer a single schema v7 for 1.10.0, with one
> migration section in the docs, and one note in the CHANGELOG, rather than
> v7 and v8 in the same release. Whichever PR is merged first, introducing
> v7, the others add their DDL to the same v7 bump.
> - Both PRs add an admin tool command deleting events in batches (purge
> events by age, purge/drain for realms). It would be good to align the
> command naming and share the batched delete logic before both are merged.
> - The schema version check is a strict equality: a 1.9.0 node fails against
> v7 schema, and a 1.10.0 node fails against v6. So, we must no mix versions
> during the upgrade. As realm_purge is pure addition (1.9.0 never touches
> it), the docs could split the migration in two steps: create the table
> while still running 1.9.0 (no impact), then bump the version. That keeps
> the downtime window to the version update and restart.
> - The time series data of realms purged before the upgrade is not in
> realm_purge, so purge/drain will never clean it. Maybe we can either state
> it explicitly in the docs, or provide an optional one-off SQL in the v7
> migration section enqueueing the realm ids present in events and the
> metrics report tables that no longer exist in entities.
>
> Regards
> JB
>
> On Thu, Nov 12, 2026 at 7:19 AM Ayush Saxena <[email protected]> wrote:
>
> > Hi all,
> >
> > PR #5089 adds a table to the relational JDBC schema, so it needs a
> > schema version bump and a migration that operators must apply before
> > starting the release that carries it. Raising it here for awareness,
> > since a required migration should not be something people meet for the
> > first time in a release note.
> >
> > The table records realms whose time-series data is still pending
> > deletion. Purging a realm today leaves its events and metrics reports
> > behind, keyed by a realm id that no longer exists, so nothing reclaims
> > them afterwards. A new admin tool command drains the queue in bounded
> > batches, rather than deleting that data inline in the purge
> > transaction, which is the statement most likely to make a purge time
> > out on a realm with a long history.
> >
> > The migration SQL and the rest of the detail are here:
> > https://github.com/apache/polaris/pull/5089
> >
> > Let me know in case anyone has any objections, feedback or better ideas.
> >
> > -Ayush
> >

Reply via email to