claudevdm opened a new pull request, #40163:
URL: https://github.com/apache/beam/pull/40163
…nvertToDataFile (#40143)
* AddFiles: per-file coverage check and pinned-column enforcement in
ConvertToDataFile
ConvertToDataFile gains a SchemaEvolutionConfig.
With options set, every Parquet file is checked against the table before its
DataFile is built, and every failure is one row on the error output:
1. FileSchemas.effective(footer) (convert plus tighten, shared with the read
side so both sides see the same schema) is computed. A footer whose schema
cannot be converted (unannotated repeated leaf, legacy MAP_KEY_VALUE, uint64,
...) yields "Could not read the file's schema: <cause>".
2. Coverage: SchemaDelta.classify against the cached table. The pre-pass
commits the schema before paths reach this DoFn, so the delta is expected to be
empty. If it is not, the table is refreshed once and classified again. A
remaining delta is reported as "Table schema does not cover the file after
refresh: <reason>", where the reason is the delta's disallowed reason when the
options forbid the change (the ROUTE_TO_ERRORS path: this is where an
incompatible file lands) or "changes not applied: ..." when the change was
allowed but somehow not committed (defensive catch all).
3. Pins: for each pinned column the table has, the file must contain it and
be provably null-free: the footer's own null count for the column
(FileSchemas.nullCount, summed over row groups) must be present and zero. A
missing count is a violation too: "cannot prove" is not "proven". Messages:
"Pinned required column X is absent from the file" / "has no null count
statistics in the file" / "has N null(s) in the file".
The pin evidence deliberately does NOT come from the Metrics object built
for the DataFile: the table's write.metadata.metrics configuration shapes those
(mode none, or the inferred-column cap on wide schemas, drops null counts
entirely), and pin enforcement must not be configurable away. Two tests pin
this down under write.metadata.metrics.default=none: a null-free pinned file
still registers and a file with nulls is still caught.
With evolution enabled, a non-Parquet file (ORC, Avro) is routed to the
error output instead of registering unchecked: none of the checks above can
read those formats, and silently registering a file the options promised to
verify would be a hole in the guarantee. With evolution disabled they register
exactly as before.
SchemaEvolutionConfig.UnverifiableFileHandling relaxes exactly the "cannot
verify" cases, never a failed check: REJECT (default) routes them to the error
output as above; ACCEPT registers a non-Parquet file unchecked, and a Parquet
file whose footer has no null count for a pinned column on trust (a writer with
statistics disabled, or a pin under a list or map, whose physical chunk path is
never mapped). An absent pinned column and a counted null stay violations under
ACCEPT, and the pin walk continues past an unproven pin so a counted one still
fails the file.
* comments
* spotbugs
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable. This will automatically add a link to the
pull request in the issue. If you would like the issue to automatically close
on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit
[https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more
information about GitHub Actions CI or the [workflows
README](https://github.com/apache/beam/blob/master/.github/workflows/README.md)
to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
