claudevdm opened a new pull request, #39933:
URL: https://github.com/apache/beam/pull/39933
ReadFooterSchema, FileScemas.canonical, CollectDistinctSchemas)
New, not yet wired code that turns a PCollection of file paths into the list
of distinct schemas those files carry, with file counts. The pre-pass that will
use it exists because manifest entries are immutable: a file registered before
the table knows one of its columns never gets stats for that column, so the
table schema has to be brought up to date before any file is registered, and
that requires looking at every footer first.
ReadFooterSchema (DoFn<String, String>) reads each Parquet footer on the
BoundedAsyncTasks pool and emits the file's canonical schema as JSON.
Non-Parquet paths and unknown extensions contribute nothing; a footer that
cannot be read or converted is logged and counted (numFooterReadErrors) but
never fails the pipeline: the per-file registration step reports such files
individually later.
FileSchemas.canonical sorts struct fields by name at every level and
renumbers ids in deterministic order.
The ids are positional and never consumed downstream: the commit side
reconciles columns by name (unionByNameWith).
CollectDistinctSchemas is a CombineFn over the canonical JSON strings
(Map<String, Long> accumulator) producing List<KV<String, Long>> ordered most
common first, ties broken by the JSON text for determinism. The most common
schema goes first because the commit side uses it as the seed when the table
does not exist yet.
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable. This will automatically add a link to the
pull request in the issue. If you would like the issue to automatically close
on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit
[https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more
information about GitHub Actions CI or the [workflows
README](https://github.com/apache/beam/blob/master/.github/workflows/README.md)
to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
