shikhabhadoria opened a new pull request, #41452: URL: https://github.com/apache/beam/pull/41452
Addresses #41356. This is 2 of 3 stacked PRs that replace #41423. Depends on #41451. Merge that first. This branch contains those commits; the commit to review here is `[Python] Add FileEnrichmentHandler for file-backed enrichment`. ## Summary - Adds `FileEnrichmentHandler` for `.parquet`, `.csv`/`.tsv`, and `.json`/`.jsonl` reference tables loaded through Beam `FileSystems`. - The lookup table is built in worker `__enter__` and shared across worker threads with `apache_beam.utils.shared.Shared`. - Supports `join_key`, `row_key` (when the element field name differs), `select_fields`, and `get_output_schema()`. - Schema peeking at pipeline construction reads the Parquet footer, or a sample of CSV/JSON rows, instead of loading the full table on the submission machine. The worker still loads the full table when the DoFn starts. - CSV decoding uses `utf-8-sig` so a UTF-8 BOM does not stick to the first column name. JSON accepts one document or an array; JSONL is parsed line by line. ## Test plan - [x] `pytest apache_beam/transforms/enrichment_handlers/file_test.py apache_beam/transforms/enrichment_test.py apache_beam/io/requestresponse_test.py` — 25 passed - [x] `ruff check` on the changed Python files - [x] `git diff --check` (no whitespace errors) - [ ] Python precommit ## Checklist - [x] If this contribution is large, please file an Apache Individual Contributor License Agreement. - [x] Mention the appropriate issue in the description (`addresses #41356`). - [x] Update `CHANGES.md` with noteworthy changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
