lostluck opened a new issue, #33650: URL: https://github.com/apache/beam/issues/33650
### What needs to happen? In 2023, Protocol Buffers replaced the "old" approach of bundling together changes via the "syntax" keyword, to a more robust feature driven approaches known as [Editions](https://protobuf.dev/editions/overview/). The main upshot, is that instead of simply having to stick to a per message or field option, it allows for a granular representation of features to be cycled in, and cycled out over time as the technology evolves. This predominantly affects the code that's generated for any given protocol buffer language, and importantly: doesn't affect the materialized form of the buffers themselves. Messages remain as backwards and forwards compatible as ever. --------- A motivation to migrating to Editions is to allow the Go Protos to use the newly released [Opaque Protocol Buffers for Go](https://go.dev/blog/protobuf-opaque) which enable future optimizations to the generated code. An initial attempt was made in #33434 but this was rolled back in #33628 due to unnoticed incompatibilities when the import to Google occured. This happened because in open source, you can set protoc to generate the new code at the tool level. However, internally to Google, they require these features to be set in the files themselves. Further the key bit is that the default 2023 Edition is *not* identical in behavior to syntax=proto3 which we currently use. ---------- The proposal is to migrate us to the current standard in Protobuf so we can take advantage of new features if needed, at any level of granularity, due to the benefits of lexical scoping. In particular, this issue is resolved when we replace the syntax = "proto3" with the appropriate features to replicate the code generation behavior of "proto3", as documented here: https://protobuf.dev/editions/features/#proto3-behavior This should require no other code changes on our part, and all SDKs should behave identically as a result, with the exception of the few places where we may need to explicitly set the field_presence type on a specific field or two, were we use `optionally` https://protobuf.dev/editions/features/#field_presence. The proposal also recommends to likely wait for the `prototiller` tool to be released, which will automatically handle this for us. ### Issue Priority Priority: 3 (nice-to-have improvement) ### Issue Components - [x] Component: Python SDK - [x] Component: Java SDK - [x] Component: Go SDK - [x] Component: Typescript SDK - [ ] Component: IO connector - [ ] Component: Beam YAML - [ ] Component: Beam examples - [ ] Component: Beam playground - [ ] Component: Beam katas - [ ] Component: Website - [ ] Component: Infrastructure - [ ] Component: Spark Runner - [ ] Component: Flink Runner - [ ] Component: Samza Runner - [ ] Component: Twister2 Runner - [ ] Component: Hazelcast Jet Runner - [ ] Component: Google Cloud Dataflow Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
