vishalmore90 opened a new pull request, #41434:
URL: https://github.com/apache/beam/pull/41434
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
- [x] Replace these instructions with a description of your change and why
you have made it.
- [x] Mention the appropriate issue in your description (for example:
`addresses #123` or `fixes #456`).
- [ ] Update `CHANGES.md` with noteworthy changes.
See
[CONTRIBUTING.md](https://github.com/apache/beam/blob/master/CONTRIBUTING.md)
for more details.
### Description
Fixes #41431
This PR resolves several critical data integrity and crash bugs in
`CloudSQLEnrichmentHandler`:
1. **Silent record loss on DB/cache miss in batch mode:** When input records
missed the database in batch mode, fewer rows were returned by the query.
Iterating only over results previously dropped unmatched input records. The
handler now matches against all input requests and yields `(request,
beam.Row())` on misses, matching single-mode semantics and
`BigQueryEnrichmentHandler`.
2. **Duplicate key loss in batch mode:** Mapping requests into a dictionary
by join key caused requests with duplicate keys in the same batch to overwrite
each other. The handler now uses a `response_map` and iterates over original
requests, ensuring all duplicate keys are enriched without dropping elements.
3. **Pipeline crash with `column_names`:** When `column_names` omitted join
columns defined in `where_clause_fields`, the SQL projection omitted the join
key, causing `create_row_key` to raise a `KeyError`. The handler now
automatically appends missing `where_clause_fields` to `column_names` during
initialization, and `create_row_key` safely handles missing keys.
4. **Cache key divergence:** Batch mode previously used `repr` while single
mode used format interpolation, causing cache lookups between single and batch
to miss for identical string keys. Both paths are now unified via a single
helper method.
### Validation
- Added comprehensive unit tests in
`sdks/python/apache_beam/transforms/enrichment_handlers/cloudsql_test.py`:
- `test_process_batch_request_handles_unmatched_and_preserves_order`
- `test_process_batch_request_handles_duplicate_keys`
- `test_column_names_auto_includes_where_clause_fields`
- `test_cache_key_consistency_between_single_and_batch`
- `test_create_row_key_missing_field_safe`
- All 23 tests in `cloudsql_test.py` pass cleanly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]