Ayush, thanks for raising this. It's critical to upgrade the schema version for any schema change. Why does this PR require a schema change? Is it due to the batch deletion? Yufei
On Wed, Nov 11, 2026 at 11:04 PM Ayush Saxena <[email protected]> wrote: > > Hi JB, > > Thanks for the review. I have incorporated the two documentation points. > > Two-step migration: the table is a pure addition that no earlier > version reads, so the docs now create it while the previous version is > still serving and record the version last, leaving the restart as the > only window where no node can serve. > > Realms purged before the upgrade: the migration section now has an > optional one-off statement that queues every realm with time-series > rows but no entities, and states what skipping it leaves behind. It > needs to run while no realm is being bootstrapped, since a realm is > indistinguishable from a purged one until its first entity is written. > > Single v7: sure. This PR introduces v7 today, and I will keep an eye > on #6013 and #4756 — if either gets merged earlier I will update this > one to add to the v7 already there. One thing to flag for that > section: #4756 does not bump the version at all right now, and CREATE > INDEX IF NOT EXISTS keeps the existing four-column index, so its new > definition needs a DROP before the CREATE. > > Naming and shared delete logic: agreed, before both merge. The loop > here is already generic over the table and its key column and takes a > cutoff, so an age-based command needs only a different cutoff, not new > deletion logic. Happy to extract it, or to adopt whatever merges > first. On naming, grouping the three under one backend-specific > parent, as the NoSQL commands are, reads better to me than three > top-level commands that work on one backend only. > > -Ayush > > On Thu, 12 Nov 2026 at 12:08, Jean-Baptiste Onofré <[email protected]> wrote: > > > > Hi Ayush, > > > > I agree that a required migration should not be a surprise in the release > > note (it something we discussed while ago when we "moved" from EclipseLink > > to JDBC afair). > > > > I'm +1 with the approach. > > > > I have a few comments though: > > - We have two PRs touching the schema. #6013 bumps the schema to v7, to add > > an index on events(realm_id, timestamp_ms), while #4756 changes the > > definition of idx_scan_report_lookup and idx_commit_report_lookup. As > > CREATE INDEX IF NOT EXISTS keeps the existing index, existing deployment > > won't get the new definition without a migration (same situation as > > idx_locations in v6). As 1.9.0 is already on vote, it means these changes > > will land in 1.10.0. I would prefer a single schema v7 for 1.10.0, with one > > migration section in the docs, and one note in the CHANGELOG, rather than > > v7 and v8 in the same release. Whichever PR is merged first, introducing > > v7, the others add their DDL to the same v7 bump. > > - Both PRs add an admin tool command deleting events in batches (purge > > events by age, purge/drain for realms). It would be good to align the > > command naming and share the batched delete logic before both are merged. > > - The schema version check is a strict equality: a 1.9.0 node fails against > > v7 schema, and a 1.10.0 node fails against v6. So, we must no mix versions > > during the upgrade. As realm_purge is pure addition (1.9.0 never touches > > it), the docs could split the migration in two steps: create the table > > while still running 1.9.0 (no impact), then bump the version. That keeps > > the downtime window to the version update and restart. > > - The time series data of realms purged before the upgrade is not in > > realm_purge, so purge/drain will never clean it. Maybe we can either state > > it explicitly in the docs, or provide an optional one-off SQL in the v7 > > migration section enqueueing the realm ids present in events and the > > metrics report tables that no longer exist in entities. > > > > Regards > > JB > > > > On Thu, Nov 12, 2026 at 7:19 AM Ayush Saxena <[email protected]> wrote: > > > > > Hi all, > > > > > > PR #5089 adds a table to the relational JDBC schema, so it needs a > > > schema version bump and a migration that operators must apply before > > > starting the release that carries it. Raising it here for awareness, > > > since a required migration should not be something people meet for the > > > first time in a release note. > > > > > > The table records realms whose time-series data is still pending > > > deletion. Purging a realm today leaves its events and metrics reports > > > behind, keyed by a realm id that no longer exists, so nothing reclaims > > > them afterwards. A new admin tool command drains the queue in bounded > > > batches, rather than deleting that data inline in the purge > > > transaction, which is the statement most likely to make a purge time > > > out on a realm with a long history. > > > > > > The migration SQL and the rest of the detail are here: > > > https://github.com/apache/polaris/pull/5089 > > > > > > Let me know in case anyone has any objections, feedback or better ideas. > > > > > > -Ayush > > >
