shikhabhadoria opened a new pull request, #41452:
URL: https://github.com/apache/beam/pull/41452

   Addresses #41356. This is 2 of 3 stacked PRs that replace #41423.
   
   Depends on #41451. Merge that first. This branch contains those commits; the 
commit to review here is `[Python] Add FileEnrichmentHandler for file-backed 
enrichment`.
   
   ## Summary
   - Adds `FileEnrichmentHandler` for `.parquet`, `.csv`/`.tsv`, and 
`.json`/`.jsonl` reference tables loaded through Beam `FileSystems`.
   - The lookup table is built in worker `__enter__` and shared across worker 
threads with `apache_beam.utils.shared.Shared`.
   - Supports `join_key`, `row_key` (when the element field name differs), 
`select_fields`, and `get_output_schema()`.
   - Schema peeking at pipeline construction reads the Parquet footer, or a 
sample of CSV/JSON rows, instead of loading the full table on the submission 
machine. The worker still loads the full table when the DoFn starts.
   - CSV decoding uses `utf-8-sig` so a UTF-8 BOM does not stick to the first 
column name. JSON accepts one document or an array; JSONL is parsed line by 
line.
   
   ## Test plan
   - [x] `pytest apache_beam/transforms/enrichment_handlers/file_test.py 
apache_beam/transforms/enrichment_test.py 
apache_beam/io/requestresponse_test.py` — 25 passed
   - [x] `ruff check` on the changed Python files
   - [x] `git diff --check` (no whitespace errors)
   - [ ] Python precommit
   
   ## Checklist
   - [x] If this contribution is large, please file an Apache Individual 
Contributor License Agreement.
   - [x] Mention the appropriate issue in the description (`addresses #41356`).
   - [x] Update `CHANGES.md` with noteworthy changes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to