tvalentyn commented on code in PR #39805: URL: https://github.com/apache/beam/pull/39805#discussion_r3813830084
########## .agent/skills/beam-concepts/SKILL.md: ########## @@ -25,6 +25,38 @@ description: Explains core Apache Beam programming model concepts including PCol ## The Beam Model Evolved from Google's MapReduce, FlumeJava, and Millwheel projects. Originally called the "Dataflow Model." +### Protos + +The `/model` directory defines the official, language-agnostic Protocol Buffer (`.proto`) and gRPC service specifications that establish the **Beam Model** and the **Beam Portability Framework**. + +#### Why `/model` Exists (Portability & Decoupling) + +Without a standardized model representation, supporting $N$ SDK languages across $M$ execution runners would require $N \times M$ separate translation layers. By defining all core pipeline concepts, data encodings, metrics, and worker RPC protocols as Protobuf messages and gRPC services, `/model` acts as the universal lingua franca: + +* **SDKs** compile user pipelines into standardized Runner API protobuf graphs. + +* **Runners** inspect, optimize, and distribute these graphs without needing SDK-specific language runtimes. + +* **Workers (SDK Harnesses)** execute user code (`DoFn`s) and communicate with runners over standardized Fn API gRPC channels. + +#### Core Directories & What They Do + +1. **`/model/pipeline` (Runner API & Core Model)**: Defines the SDK- and runner-independent representation of pipelines (`Pipeline`, `Components`, `PTransform`, `PCollection`, `Coder`), timestamps/constants, Beam Schemas (`Row`, `Field`), and execution metrics (`MonitoringInfo`). +2. **`/model/fn-execution` (Fn API & Provisioning)**: Defines bidirectional gRPC services between runners and worker SDK harnesses for bundle execution (`Control`), element streaming (`Data`), state/timer access (`State`), log forwarding (`Logging`), and container initialization (`Provisioning`). +3. **`/model/job-management` (Job, Expansion, & Artifact APIs)**: Defines gRPC interfaces for submitting and monitoring jobs on remote servers (`JobService`), resolving cross-language transforms in remote SDKs (`ExpansionService`), and staging dependency artifacts or container images (`ArtifactService`). +4. **`/model/interactive` (Interactive API)**: Defines metadata and stream headers for recording and replaying data in Interactive Beam notebooks. + +#### What Agents Need to Pay Special Attention To + +* **Conservative Proto Changes**: Proto changes are generally conservative and accepted only when there is a compelling reason and community consensus. Every addition introduces a new obligation that each SDK and runner must support; adding new Beam model elements (portable types, capabilities) increases the compatibility gap between SDK and runner capabilities. +* **URNs Are the API Contract**: Transforms, coders, windowing strategies, environments, and metrics are bound together by standardized string URNs (e.g., `beam:transform:pardo:v1`, `beam:coder:bytes:v1`). When inspecting or creating transforms across languages, always verify URN mappings and registry handlers in both the SDK and Runner runtimes. +* **Strict Backward & Wire Compatibility**: + * Never renumber, delete, or modify existing field IDs or URN strings in `.proto` files, as they are used across distributed RPC boundaries and persisted checkpoints. + * When extending Java/Python SDK classes to support new proto fields, use **Builder patterns** (e.g., `OutputBuilder`) rather than adding arguments to legacy static factory methods (`WindowedValue.of(...)`), which breaks API compatibility. Review Comment: I think for Python using Builders is not as common and compatibility can be achieved by using optional kwargs with default values; builders might create overdesigned code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
