vishalmore90 opened a new pull request, #41239: URL: https://github.com/apache/beam/pull/41239
Fixes #41238 ### Description In Beam YAML, `YamlProviders.Flatten._merge_schemas()` handles schema unification when flattening multiple `PCollection`s. Previously: 1. Field names were collected using `set().union(*(s.keys() for s in schemas))`, causing non-deterministic schema field ordering across Python runs. 2. Mixing optional and non-optional types of the same inner type (e.g. `int` and `Optional[int]`) was treated as a type collision and degraded the unified field type to `Optional[Any]`. 3. Fields were unconditionally typed as `Optional[...]`, even when present and required across all input schemas. 4. Input `PCollection`s with identical schemas were still subjected to synthetic schema creation and per-element row conversion via `_unify_element_with_schema`. This PR fixes these issues by: - Returning `None` early from `_merge_schemas` if all `PCollection`s share identical schemas, allowing `beam.Flatten` to execute natively without row conversion overhead. - Using `dict.fromkeys(...)` to preserve deterministic field ordering. - Unwrapping optional types using `_match_is_optional` and `extract_optional_type` to unify types with compatible bases (e.g., `int` + `Optional[int]` -> `Optional[int]`). - Keeping fields non-optional when they are present and non-optional across all input schemas. ### Checklist - [x] Description of change and rationale included above. - [x] Mentioned the appropriate issue (`Fixes #41238`). - [x] Updated `CHANGES.md` under `## Bugfixes`. - [x] Added unit tests covering identical schemas, deterministic ordering, optional unwrapping, required field preservation, and conflicting type fallback. - [x] Verified code formatting with `yapf` and `pycodestyle`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
