claudevdm opened a new pull request, #40104:
URL: https://github.com/apache/beam/pull/40104
The commit side of the schema pre-pass for an existing table. Takes one
window's distinct canonical file schemas (most common first) and evolves the
table once to cover them all, adding exactly one schema version per window
regardless of how many distinct schemas it held.
Flow, per attempt:
1. Load + classify: fresh table load; SchemaDelta.classify per schema. Empty
deltas skip, deltas the config disallows go to the incompatible list with their
reason.
2. Fold on a scratch transaction that is never committed: per accepted
schema, one unionByNameWith plus explicit relaxations for absent required
columns. Two schemas can conflict with each other even when each is fine
against the table (score long vs score string); that surfaces only while
staging, so the offender is blamed ("conflicts with another file schema in the
same window") and the fold is rebuilt without it, most common first, so the
majority wins ties.
3. Relax new fields: the union keeps the file's optionality below newly
added columns, so one lucky file would impose required fields on everyone.
Every field new in this window becomes optional, at every level including under
lists and maps; only map-key subtrees are kept (keys are required by
definition). Pins deliberately do not shape new columns,they keep existing
required columns from being relaxed (enforced in classify) and gate files at
registration.
4. Replay + commit: a single unionByNameWith(foldedSchema) on the real
transaction reproduces the fold's net effect (a checkState asserts they agree),
the name mapping is regenerated if absent/stale (zero-copy files are unreadable
without it, reason enough to commit alone), then one commit. Under
FAIL_PIPELINE, incompatible schemas throw before anything commits; under
ROUTE_TO_ERRORS they're logged and their files fail the per-file check later
with the same reason. Nothing commits when nothing changed.
Retry: CommitFailedException restarts from the load, up to 5 attempts,
thrown by the committer on a concurrent commit, and by a guard on every
transaction creation (Iceberg refreshes per newTransaction(), so mid-attempt
concurrent schema changes are detected as drift from the classified snapshot
and retried rather than failed). Classification is redone against the new
state, not replayed.
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable. This will automatically add a link to the
pull request in the issue. If you would like the issue to automatically close
on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit
[https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more
information about GitHub Actions CI or the [workflows
README](https://github.com/apache/beam/blob/master/.github/workflows/README.md)
to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
