vishalmore90 opened a new pull request, #41434:
URL: https://github.com/apache/beam/pull/41434

   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
    - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
    - [x] Replace these instructions with a description of your change and why 
you have made it.
    - [x] Mention the appropriate issue in your description (for example: 
`addresses #123` or `fixes #456`).
    - [ ] Update `CHANGES.md` with noteworthy changes.
   
   See 
[CONTRIBUTING.md](https://github.com/apache/beam/blob/master/CONTRIBUTING.md) 
for more details.
   
   ### Description
   Fixes #41431
   
   This PR resolves several critical data integrity and crash bugs in 
`CloudSQLEnrichmentHandler`:
   1. **Silent record loss on DB/cache miss in batch mode:** When input records 
missed the database in batch mode, fewer rows were returned by the query. 
Iterating only over results previously dropped unmatched input records. The 
handler now matches against all input requests and yields `(request, 
beam.Row())` on misses, matching single-mode semantics and 
`BigQueryEnrichmentHandler`.
   2. **Duplicate key loss in batch mode:** Mapping requests into a dictionary 
by join key caused requests with duplicate keys in the same batch to overwrite 
each other. The handler now uses a `response_map` and iterates over original 
requests, ensuring all duplicate keys are enriched without dropping elements.
   3. **Pipeline crash with `column_names`:** When `column_names` omitted join 
columns defined in `where_clause_fields`, the SQL projection omitted the join 
key, causing `create_row_key` to raise a `KeyError`. The handler now 
automatically appends missing `where_clause_fields` to `column_names` during 
initialization, and `create_row_key` safely handles missing keys.
   4. **Cache key divergence:** Batch mode previously used `repr` while single 
mode used format interpolation, causing cache lookups between single and batch 
to miss for identical string keys. Both paths are now unified via a single 
helper method.
   
   ### Validation
   - Added comprehensive unit tests in 
`sdks/python/apache_beam/transforms/enrichment_handlers/cloudsql_test.py`:
     - `test_process_batch_request_handles_unmatched_and_preserves_order`
     - `test_process_batch_request_handles_duplicate_keys`
     - `test_column_names_auto_includes_where_clause_fields`
     - `test_cache_key_consistency_between_single_and_batch`
     - `test_create_row_key_missing_field_safe`
   - All 23 tests in `cloudsql_test.py` pass cleanly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to