Personally, I do not mind the builder pattern or dealing with getters/setters; if that is the answer, I am OK with it, my primary concern is when things are combined and I can construct broken messages (bypassing builder, and setters) and the builder pattern.
Extra context, I am optimizing for LLM effectiveness, after doing some polyglot work around TypeScript, Go, and Rust. I found that Rust type system avoid hallucination to a point that isn't easy to ignore. For that to happen, the types coming out of the protobuf and its API needs to avoid silly mistakes. The same mistakes that some developer make based on suboptimal assumptions (context). Without sidetracking too much, the biggest pain in the Go version is that everything is a pointer; we can not know if the reason for the pointer is because we actually want `nil` values or because of memory layout or technical concern. I wish an Option type existed in Go or added to the protobuf for this same reason. And well, defaults .... I digress. I can strategize around the package, in the worst case, hoping that the protoc plugin allows some policies to change the behaviour. The interesting situation is that, reading about the required deprecation a bit more, definitely wouldn't like stale proto files to blow up if new information is added but my Processor does't care since it doesn't make use of the info, and therefore, it doesn't need the new proto file. Purely speaking about Event Sourcing so my events are immutable, never breaking change messages. What do you recommend me to do here? I need to find a way to test the v4 asap, I feel it is prudent to move to it On Tuesday, June 3, 2025 at 8:29:06 AM UTC-4 Em Rauch wrote: > > but I would expect two types based on my needs (PartialMessage<Message> > and Message). > > I guess I am going against what the philosophy is, which it is OK by me, > just feel suboptimal, I want to get the most I can out of the type system > without requiring that much discipline from developers. > > This does make perfect sense in general: if required was newly created in > 2025 it maybe could try to have semantics of always having Builders, being > able to parsePartial into Builders. Note that not many of the official > implementations of Protobuf do not use builder pattern though, so anything > like that would be very difficult to retrofit. > > But the context is more that the idea of "blanket enforcement of presence > of a field at parse time" is considered a misfeature (as discussed in the > "Required is Strongly Deprecated" section here > <https://protobuf.dev/programming-guides/proto2/#:~:text=Required%20is%20Strongly%20Deprecated>). > > Under the expected usecase patterns of Protobuf, its considered a big > footgun since many people believe they have a field that is obviously > always strictly required, but actually a few years later they will want to > replace it with something different, and hard enforcement at parse time > makes that evolution almost impossible. > > Zooming out a bit: for the usecase you're describing, for the official > Google supported implementations, I think just doing the normal / natural > thing should be able to achieve the effective behavior you're thinking > about. Make a normal message, make it have normal message-typed fields, > treat it as normal child structs which are just "always there". Treat the > maybe-lazy-allocation of those inner structs as an implementation detail, > and treat the has-bit tracking as an implementation detail. If the > default-initialized child struct isn't a valid value, check in your > application code against that condition, rather than checking against the > hassers (it would practically never matter for "this is a valid value of > the child" if the child message field was set but in turn none of the > fields on it were set anyway). > > With Prost specifically, it may be harder to achieve that pattern > naturally since all of the message-typed fields are typed as Option<>, but > thats the nature of it and there's some tradeoffs Prost makes in turn for > other ergonomic benefits. The V4 official Rust impl is in beta right now, > it is an opaque api instead of open structs and so doesn't have the same > implication, but it does end up being a design choice that will come with a > number of other tradeoffs (some of which are already detailed in our beta > documentation here <https://protobuf.dev/reference/rust/>). > > On Mon, Jun 2, 2025 at 5:05 PM Yordis Prieto <yordis...@gmail.com> wrote: > >> > Can you clarify a bit more what it is you are trying to achieve, and if >> there are any official implementations where Option/nullability is a pain >> point? >> >> To be clear, I haven't use the official Rust package. I would love to, >> the conversation started in Prost, and the short comings there. I am trying >> to figure out where to find the official v4 rust version to figure out the >> situation there. >> >> > I think it could not omit the Option<> on required message fields >> because the semantic of required is inherently only on wire and not >> in-memory: >> >> I figured, that is where I wish I couldn't even construct a "broken" >> type, if a given field is required, then it must be set in order to >> construct the message. Otherwise, the type system isn't helping much and we >> must figure out the issues at runtime. I am unsure of the details about >> parsePartial and memory vs. wire; but I would expect two types based on my >> needs (PartialMessage<Message> and Message). >> >> I guess I am going against what the philosophy is, which it is OK by me, >> just feel suboptimal, I want to get the most I can out of the type system >> without requiring that much discipline from developers. >> >> On Monday, June 2, 2025 at 4:00:07 PM UTC-4 Em Rauch wrote: >> >>> Can you clarify a bit more what it is you are trying to achieve, and if >>> there are any official implementations where Option/nullability is a pain >>> point? >>> >>> So in general the official Google Protobuf implementations do not expose >>> Option<SomeMessage> or nullable-SomeMessage* in almost any of our >>> implementations (as per the page I mentioned on the github issue: >>> https://protobuf.dev/design-decisions/nullable-getters-setters/) >>> >>> If I'm following collectly, on all official implementations you should >>> be able to get the behavior you're describing by just declaring the child >>> messages and just don't check their has-bits. As far as the observable >>> behavior is concerned all of the submessages being eagerly allocated or >>> lazily allocated should be an implementation detail. >>> >>> proto2 `required` is considered strongly regretted and we discourage its >>> use, but just considering the semantics of required with Prost shape of >>> API, I think it could not omit the Option<> on required message fields >>> because the semantic of required is inherently only on wire and not >>> in-memory: its the normal design that you would first construct a message >>> without the required fields set, set them, and then eventually serialize >>> it. Even in terms of data that was from parsed data, most Google official >>> Protobuf implementations support a "parsePartial" which means "parse these >>> bytes but don't enforce required, and then me write the logic against >>> hassers directly". >>> >>> >>> On Mon, Jun 2, 2025 at 3:46 PM Yordis Prieto <yordis...@gmail.com> >>> wrote: >>> >>>> As requested by [@esrauchg](https://github.com/esrauchg), moving the >>>> conversation here: >>>> https://github.com/tokio-rs/prost/issues/1031#issuecomment-2932003751. >>>> >>>> Although this originated in a `prost` discussion, the underlying issue >>>> concerns >>>> how the official protobuf toolchain treats field presence. The intent >>>> here is to >>>> clarify the expected stance and behavior of the generator. >>>> >>>> **What language does this apply to?** >>>> >>>> Primarily **Rust**, but also interested in **Go** and **TypeScript** >>>> as they >>>> relate to protobuf code generation. >>>> >>>> **Describe the problem you are trying to solve.** >>>> >>>> The generated types are structurally imprecise: **all fields are >>>> wrapped in `Option<T>`**, >>>> even those marked as `required` (explicitly or implicitly) in the proto >>>> definition. >>>> >>>> This leads to: >>>> >>>> - Verbose boilerplate (`Some(...)` wrappers) >>>> - Increased cognitive overhead >>>> - Risk of runtime errors when unwrapping values that should be >>>> guaranteed >>>> >>>> ```rust >>>> fn calculate_minimum_bid_increment(cmd: &StartAuctionRun) -> Result< >>>> MoneyAmount, Error> { >>>> match &cmd.minimum_bid_increment_policy { >>>> Some(policy) => match &policy.policy { >>>> Some(minimum_bid_increment_policy::Policy::Fixed(fixed_policy)) => { >>>> //... >>>> } >>>> // ... >>>> // NOTE: this should never happen, it is required >>>> None => Err(Error::MinimumBidIncrementPolicyRequired), >>>> }, >>>> // NOTE: this should never happen, it is required >>>> None => Err(Error::MinimumBidIncrementPolicyRequired), >>>> } >>>> } >>>> ``` >>>> >>>> Despite the domain clearly requiring this field, the type system does >>>> not >>>> enforce it. Structurally speaking. >>>> >>>> We're using protobuf as the canonical schema for all: >>>> >>>> - Commands >>>> - Events >>>> - Aggregate Snapshot >>>> >>>> That applies across multiple language runtimes (via WASM modules) and is >>>> critical for: >>>> >>>> - Schema evolution and change detection >>>> - Consistent interop across services >>>> - Serialization correctness >>>> - Avoiding runtime reflection or manual encoders/decoders >>>> >>>> We treat protobuf-generated types as the source of truth and only >>>> validate >>>> `commands` post-deserialization (or via protovalidate extension). >>>> >>>> Here is the existing callbacks (thus far): >>>> >>>> ```rust >>>> pub type InitialState<State> = fn() -> State; >>>> pub type IsTerminal<State> = fn(state: State) -> bool; >>>> pub type Decide<State, Command, Event, Error> = >>>> fn(state: &State, command: Command) -> Result<Decision<State, Command, >>>> Event, Error>, Error>; >>>> pub type Evolve<State, Event> = fn(state: &State, event: Event) -> >>>> State; >>>> >>>> pub type GetStreamId<Command> = fn(Command) -> String; >>>> pub type IsOrigin<Event> = fn(Event) -> bool; >>>> pub type GetEventID<Event> = fn(Event) ->String; >>>> ``` >>>> >>>> **Describe the solution you'd like** >>>> >>>> The code generator should respect the `required` field modifier in >>>> `.proto` >>>> definitions, emitting non-optional Rust fields where appropriate. >>>> >>>> That would: >>>> >>>> - Better align with schema intent >>>> - Eliminate unnecessary `Option<T>` wrappers >>>> - Improve safety and ergonomics >>>> >>>> **Describe alternatives you've considered** >>>> >>>> Creating application-level copies of protobuf types, but such types >>>> have very little value in our context. The distintion between >>>> application >>>> and serialization matters better little to us, especially when >>>> protobuffer >>>> files can not break change. >>>> >>>> **Additional context** >>>> >>>> - https://github.com/tokio-rs/prost/pull/1286 >>>> - https://github.com/tokio-rs/prost/issues/1031 >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Protocol Buffers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to protobuf+u...@googlegroups.com. >>>> To view this discussion visit >>>> https://groups.google.com/d/msgid/protobuf/b5386744-e770-430e-bccb-959409a05641n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/protobuf/b5386744-e770-430e-bccb-959409a05641n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to protobuf+u...@googlegroups.com. >> > To view this discussion visit >> https://groups.google.com/d/msgid/protobuf/41ab96b4-4af9-4df2-8d88-fdb9f469f70dn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/protobuf/41ab96b4-4af9-4df2-8d88-fdb9f469f70dn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/protobuf/faa787ad-543f-4132-b6a7-edd4d8fe6572n%40googlegroups.com.
