jk-shah opened a new issue, #41355:
URL: https://github.com/apache/beam/issues/41355
### Description
Beam YAML currently exposes `MLTransform` (`yaml_ml.py`) for
embeddings/preprocessing and `Enrichment` (`yaml_enrichment.py`) for feature
lookups, but has **no declarative `RunInference` provider** for scoring records
in a YAML pipeline using local HuggingFace models, Vertex AI endpoints, or
custom Python `ModelHandler`s.
### Proposed YAML Syntax
```yaml
pipeline:
transforms:
- type: RunInference
input: ReadFromPubSub
config:
provider: HuggingFacePipeline # or VertexAI, PyTorch, Sklearn,
Custom
model_config:
task: text-classification
model: distilbert/distilbert-base-uncased-finetuned-sst-2-english
large_model: true
input_field: comment_text
output_field: sentiment # or flatten_predictions: true ->
sentiment_label, sentiment_score
min_batch_size: 8
max_batch_size: 64
error_handling:
output: failed_inferences
```
### Key Capabilities
1. **Lazy Worker-Side Loading**: Defer model downloads and endpoint client
initialization to worker `setup()` / `load_model()` so YAML graph validation
(`--dry_run`) remains offline and fast.
2. **Schema-Preserving Output**: Attach predictions onto the input
`beam.Row` (via `KeyedModelHandler`) with concrete output `RowTypeConstraint`s
so downstream `Filter`, `Combine`, and `WriteToParquet` stages type-check
cleanly.
3. **Per-Element `error_handling` DLQ & Poison-Pill Isolation**: Integrate
with Beam YAML's standard `error_handling` strip-and-route contract, falling
back to per-element retry when a single malformed record fails a batch of `N`
so `N-1` valid records succeed without evicting/reloading the in-memory model.
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2/P3)
### Issue Components
- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [x] Component: Beam YAML
- [x] Component: Beam ML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]