jk-shah opened a new issue, #41353:
URL: https://github.com/apache/beam/issues/41353

   ### Description
   
   In `sdks/python/apache_beam/yaml/yaml_provider.py` (lines `1160-1186`), Beam 
YAML's `windowing` / `WindowInto` parser currently leaves `# TODO: Triggering, 
etc.` unimplemented and raises `ValueError: Unknown parameters` if 
`allowed_lateness`, `trigger`, or `accumulation_mode` is specified:
   
   ```python
   if spec:
     raise ValueError(f'Unknown parameters {spec.keys()}')
   # TODO: Triggering, etc.
   return beam.WindowInto(window_fn)
   ```
   
   Without these parameters, streaming Beam YAML pipelines cannot:
   1. Configure `allowed_lateness` (any late event arriving after the watermark 
passes `window.end` is silently dropped by `GroupByKey` with 
`allowed_lateness=0`),
   2. Configure early/speculative or late-firing triggers 
(`AfterWatermark(early=..., late=...)`, `Repeatedly(AfterProcessingTime(...))`, 
`AfterCount(...)`), or
   3. Choose between `ACCUMULATING` and `DISCARDING` pane accumulation modes 
(and expose pane metadata when needed).
   
   ### Proposed YAML Syntax
   
   ```yaml
   pipeline:
     transforms:
       - type: WindowInto
         input: AssignTimestamps
         windowing:
           type: fixed
           size: 60s
           allowed_lateness: 300s
           accumulation_mode: discarding   # or accumulating
           trigger:
             type: after_watermark
             early:
               type: after_processing_time
               delay: 10s
             late:
               type: after_count
               count: 1
   ```
   
   ### Key Capabilities
   1. Parse duration strings (`300s`, `5m`, `1h`) for `allowed_lateness`.
   2. Support declarative `trigger` specs mapping to `AfterWatermark`, 
`AfterProcessingTime`, `AfterCount`, `AfterAny`, `AfterAll`, and `Repeatedly`.
   3. Map `accumulation_mode` to `AccumulationMode.ACCUMULATING` / 
`AccumulationMode.DISCARDING` (requiring an explicit `accumulation_mode` or 
safe default when multi-firing triggers are configured).
   
   ### Issue Priority
   
   Priority: 2 (default / most feature requests should be filed as P2/P3)
   
   ### Issue Components
   
   - [x] Component: Python SDK
   - [ ] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [ ] Component: IO connector
   - [x] Component: Beam YAML
   - [ ] Component: Beam ML
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Infrastructure
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Prism Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to