shikhabhadoria opened a new pull request, #41423:
URL: https://github.com/apache/beam/pull/41423

   ## Summary
   
   Addresses #41356.
   
   This PR adds a file-backed lookup handler (`FileEnrichmentHandler`) and 
`on_miss`, `prefix`, and `error_handling` dead-letter routing semantics to 
`Enrichment` (`apache_beam.transforms.enrichment` and 
`apache_beam.yaml.yaml_enrichment`):
   
   1. **`FileEnrichmentHandler` 
(`apache_beam.transforms.enrichment_handlers.file`)**:
      - Loads `.parquet`, `.csv`/`.tsv`, and `.json`/`.jsonl` reference tables 
via Beam `FileSystems` in worker `__enter__`, sharing the in-memory lookup 
dictionary across worker threads using `apache_beam.utils.shared.Shared`.
      - Supports `join_key` (and `row_key`), `select_fields` projection, and 
static output schema inference (`get_output_schema()`).
   2. **`on_miss` & `prefix` Controls 
(`apache_beam.transforms.enrichment.Enrichment`)**:
      - Adds `on_miss` modes:
        - `'null'` (default): left-join semantics, populating known enriched 
fields with `None` when a key is not found.
        - `'drop'`: inner-join semantics, dropping unmatched elements.
        - `'dlq'`: routes unmatched elements to the configured dead-letter 
output when exception handling is enabled.
        - `'fail'`: raises a `KeyError` on missing keys to fail the pipeline.
      - Adds `prefix` to `cross_join` and `Enrichment` to prefix enriched 
column names and avoid silent field collisions.
      - Infers output `RowTypeConstraint` when the source handler provides a 
static schema.
   3. **Beam YAML `error_handling` Integration 
(`apache_beam.yaml.yaml_enrichment` & `RequestResponseIO`)**:
      - Registers `'File': FileEnrichmentHandler` in 
`yaml_enrichment.enrichment_transform`.
      - Decorates `enrichment_transform` with 
`@maybe_with_exception_handling_transform_fn` and adds 
`with_exception_handling` support to `RequestResponseIO` and `Enrichment` so 
both RPC timeouts/failures and `on_miss: dlq` misses are routed to the standard 
Beam YAML `error_handling` dead-letter output.
   
   ## Checklist
   
    - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
    - [x] Replace these instructions with a description of your change and why 
you have made it.
    - [x] Mention the appropriate issue in your description (for example: 
`addresses #123` or `fixes #456`).
    - [x] Update `CHANGES.md` with noteworthy changes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to