Hi Ayush,

I agree that a required migration should not be a surprise in the release
note (it something we discussed while ago when we "moved" from EclipseLink
to JDBC afair).

I'm +1 with the approach.

I have a few comments though:
- We have two PRs touching the schema. #6013 bumps the schema to v7, to add
an index on events(realm_id, timestamp_ms), while #4756 changes the
definition of idx_scan_report_lookup and idx_commit_report_lookup. As
CREATE INDEX IF NOT EXISTS keeps the existing index, existing deployment
won't get the new definition without a migration (same situation as
idx_locations in v6). As 1.9.0 is already on vote, it means these changes
will land in 1.10.0. I would prefer a single schema v7 for 1.10.0, with one
migration section in the docs, and one note in the CHANGELOG, rather than
v7 and v8 in the same release. Whichever PR is merged first, introducing
v7, the others add their DDL to the same v7 bump.
- Both PRs add an admin tool command deleting events in batches (purge
events by age, purge/drain for realms). It would be good to align the
command naming and share the batched delete logic before both are merged.
- The schema version check is a strict equality: a 1.9.0 node fails against
v7 schema, and a 1.10.0 node fails against v6. So, we must no mix versions
during the upgrade. As realm_purge is pure addition (1.9.0 never touches
it), the docs could split the migration in two steps: create the table
while still running 1.9.0 (no impact), then bump the version. That keeps
the downtime window to the version update and restart.
- The time series data of realms purged before the upgrade is not in
realm_purge, so purge/drain will never clean it. Maybe we can either state
it explicitly in the docs, or provide an optional one-off SQL in the v7
migration section enqueueing the realm ids present in events and the
metrics report tables that no longer exist in entities.

Regards
JB

On Thu, Nov 12, 2026 at 7:19 AM Ayush Saxena <[email protected]> wrote:

> Hi all,
>
> PR #5089 adds a table to the relational JDBC schema, so it needs a
> schema version bump and a migration that operators must apply before
> starting the release that carries it. Raising it here for awareness,
> since a required migration should not be something people meet for the
> first time in a release note.
>
> The table records realms whose time-series data is still pending
> deletion. Purging a realm today leaves its events and metrics reports
> behind, keyed by a realm id that no longer exists, so nothing reclaims
> them afterwards. A new admin tool command drains the queue in bounded
> batches, rather than deleting that data inline in the purge
> transaction, which is the statement most likely to make a purge time
> out on a realm with a long history.
>
> The migration SQL and the rest of the detail are here:
> https://github.com/apache/polaris/pull/5089
>
> Let me know in case anyone has any objections, feedback or better ideas.
>
> -Ayush
>

Reply via email to