claudevdm opened a new pull request, #40062:
URL: https://github.com/apache/beam/pull/40062
The classifier behind the options. Iceberg's unionByNameWith has no knobs:
it adds, relaxes and promotes in one go, or throws. To honour
ALLOW_FIELD_ADDITION / ALLOW_FIELD_RELAXATION / ALLOW_TYPE_PROMOTION
separately, SchemaDelta.classify(table, fileSchema) applies the union on a
throwaway UpdateSchema (apply(), never commit()), diffs the result against the
current table schema, and labels every change:
- FIELD_ADDITION: a field id present only after the union.
- FIELD_RELAXATION: required before, optional after; or a required table
column with no counterpart in the file at all (see below).
- TYPE_PROMOTION: same id, wider primitive type after.
- CONFLICT: anything else. The union throwing (ValidationException or
IllegalArgumentException, e.g. int column vs string file column, a dotted or
empty file column name), a field removed by the union (cannot happen with
unionByName but is refused rather than trusted), a struct where a primitive
was, a doc string or default changing, a promotion Iceberg would not allow
(TypeUtil.isPromotionAllowed guard, so a bad union result is never staged as a
"promotion").
The diff is keyed by field id and walks fields attribute by attribute (name,
optionality, type kind, doc, defaults), so an attribute the union silently
changes is reported rather than committed unnoticed. Changes are listed in a
deterministic order (unquoted path) with quoted names in messages so a reviewer
can find them in the schema.
Absence rule: a required table column that the file lacks is a
FIELD_RELAXATION, not a pass. Registering such a file would put nulls in a
required column for every reader; the fix is to relax the column explicitly
(the commit side stages makeColumnOptional for exactly the paths
absentRequiredPaths() reports). The walk descends through structs whose parent
is present, through list elements and map values (paths use "element" and
"value", which makeColumnOptional accepts); an absent struct is itself the
relaxation, its children are not listed separately; map keys are required by
definition and skipped.
Pins: Change.allowedBy(config) refuses a relaxation of a pinned path even
when ALLOW_FIELD_RELAXATION is set, and disallowedReason(config) names it, so
"id is pinned" shows up as the reason rather than a generic "relaxation not
allowed".
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable. This will automatically add a link to the
pull request in the issue. If you would like the issue to automatically close
on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit
[https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more
information about GitHub Actions CI or the [workflows
README](https://github.com/apache/beam/blob/master/.github/workflows/README.md)
to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
