claudevdm opened a new pull request, #40167:
URL: https://github.com/apache/beam/pull/40167
AddFiles: `dry_run` reports what the schema pre-pass would do
`SchemaEvolutionConfig.setDryRun(true)` (`dry_run: true` in YAML) turns
AddFiles into a report-only transform. The read side runs as usual (footers,
distinct schemas); instead of committing and registering, a new
`dry_run_report` output gets one row per distinct file schema plus a `create`
row, marker rows for unreadable and ORC/Avro files, and a summary row.
`snapshots` and `errors` are empty. The output only exists when `dry_run` is
set, so existing YAML pipelines are unaffected.
The diff has two halves. Reading them in this order is easiest.
**1. `CommitSchemaUnion`: the commit now plans before it writes** (about a
third of the diff, plus `SchemaDelta` and `CommitSchemaUnionTest`).
Everything a commit decides is pulled into `Plan`, built by `plan()`: load
the table (or note it is missing), classify and fold on scratch transactions
(or fold for creation), then the configuration checks against the schema the
run would end with. `commitOnce` and `create` consume the plan; `DryRunReport`
turns the same plan into rows. A check added to the plan reaches both, so the
report cannot drift from the commit. In particular a schema that is fine
against the table but conflicts with another schema of the input is reported
with the blame a real run assigns.
**2. The dry run itself**: `DryRunReport` (new), the `expand` wiring in
`AddFiles`, the `dry_run` knob in `SchemaEvolutionConfig` and the YAML
provider, the marker keys in `ReadFooterSchema`, the transform-level tests in
`AddFilesTest`, and a YAML fixture that runs a dry run against a missing table,
writes the report to JSON and asserts the full rows.
### Report rows
| column | meaning |
|---|---|
| `schema_key` | short murmur3 key of the schema JSON; `create`, `unread`,
`unchecked`, `summary` for the other rows |
| `schema` | canonical schema JSON; the union the table would be created
with on the `create` row |
| `num_files` | files carrying it |
| `changes` | the `SchemaDelta` descriptions; one `create optional <name>
<type>` per column on the `create` row; the totals line plus any table-level
change (name mapping) on the summary |
| `allowed` | whether a real run would accept it |
| `reason` | why not, else empty; the consequence on the summary |
| `would_create_table` | false whenever a real run would not create the
table, including when it would fail first |
| `summary` | true on the summary row |
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable. This will automatically add a link to the
pull request in the issue. If you would like the issue to automatically close
on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit
[https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more
information about GitHub Actions CI or the [workflows
README](https://github.com/apache/beam/blob/master/.github/workflows/README.md)
to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
