claudevdm opened a new pull request, #40124:
URL: https://github.com/apache/beam/pull/40124
Before this change the pre-pass required an existing table. AddFiles without
evolution creates a missing table from whichever file happens to reach
ConvertToDataFile first; with the pre-pass every schema of the window is known
before anything is created.
Create path (CommitSchemaUnion.create), taken when loadTable throws
NoSuchTableException:
- The window's schemas are folded into one union on a scratch create
transaction that is never committed (the same staging loop as the evolve path,
so a schema conflicting with another is reported or skipped per the handling,
and the create is abandoned under FAIL_PIPELINE before anything exists).
- The real table is built directly from createdSchema(union, config): every
column optional at every level - one lucky file's declared or proven
required-ness must not become a table constraint that the next file violates -
except pinned paths and their ancestors, which are created required. Creation
is the schema-authoring moment; evolution never tightens columns afterwards. A
pin that no file schema carries fails the create under FAIL_PIPELINE (later
windows could only add the column optional, so the pin would stay inert
forever) and warns under ROUTE_TO_ERRORS. The created table is born with a
single schema version.
- Creation is not gated on any particular evolution option: the options
guard an existing table's schema, and there is none to guard yet.
- Partition spec, sort order and table properties come from TableCreation
(the AddFiles constructor arguments), resolved against the union, so a
partition or sort column carried by any schema of the window works; the name
mapping is written in the same transaction.
- Create race: two workers (or a concurrent non-evolution AddFiles) may
create the table at once. AlreadyExistsException joins CommitFailedException in
the retry, and the next attempt takes the evolve path against the table the
other party created.
- Empty window against a missing table returns NO_TABLE (-1) and creates
nothing.
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable. This will automatically add a link to the
pull request in the issue. If you would like the issue to automatically close
on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit
[https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more
information about GitHub Actions CI or the [workflows
README](https://github.com/apache/beam/blob/master/.github/workflows/README.md)
to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
