jk-shah opened a new issue, #41353:
URL: https://github.com/apache/beam/issues/41353
### Description
In `sdks/python/apache_beam/yaml/yaml_provider.py` (lines `1160-1186`), Beam
YAML's `windowing` / `WindowInto` parser currently leaves `# TODO: Triggering,
etc.` unimplemented and raises `ValueError: Unknown parameters` if
`allowed_lateness`, `trigger`, or `accumulation_mode` is specified:
```python
if spec:
raise ValueError(f'Unknown parameters {spec.keys()}')
# TODO: Triggering, etc.
return beam.WindowInto(window_fn)
```
Without these parameters, streaming Beam YAML pipelines cannot:
1. Configure `allowed_lateness` (any late event arriving after the watermark
passes `window.end` is silently dropped by `GroupByKey` with
`allowed_lateness=0`),
2. Configure early/speculative or late-firing triggers
(`AfterWatermark(early=..., late=...)`, `Repeatedly(AfterProcessingTime(...))`,
`AfterCount(...)`), or
3. Choose between `ACCUMULATING` and `DISCARDING` pane accumulation modes
(and expose pane metadata when needed).
### Proposed YAML Syntax
```yaml
pipeline:
transforms:
- type: WindowInto
input: AssignTimestamps
windowing:
type: fixed
size: 60s
allowed_lateness: 300s
accumulation_mode: discarding # or accumulating
trigger:
type: after_watermark
early:
type: after_processing_time
delay: 10s
late:
type: after_count
count: 1
```
### Key Capabilities
1. Parse duration strings (`300s`, `5m`, `1h`) for `allowed_lateness`.
2. Support declarative `trigger` specs mapping to `AfterWatermark`,
`AfterProcessingTime`, `AfterCount`, `AfterAny`, `AfterAll`, and `Repeatedly`.
3. Map `accumulation_mode` to `AccumulationMode.ACCUMULATING` /
`AccumulationMode.DISCARDING` (requiring an explicit `accumulation_mode` or
safe default when multi-firing triggers are configured).
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2/P3)
### Issue Components
- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [x] Component: Beam YAML
- [ ] Component: Beam ML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]