pjfanning opened a new pull request, #3500: URL: https://github.com/apache/pekko/pull/3500
### Motivation Several of the remoting wire formats carry a payload whose enclosed message is itself a serialized payload — `Some`, `Optional`, `Status.Failure`, `StatusReply`, a `Throwable` cause, or an `ActorSelectionMessage`. Each level is parsed on its own, so neither the protobuf parser's own nesting limit nor the size of the message bounds how deep the chain can go: the recursion tracks the nesting rather than the number of bytes. A sufficiently deep chain fails with a `StackOverflowError` rather than a serialization error, which is not how the rest of the deserialization path reports a message it cannot read. ### Modification Add a per-thread nesting-depth counter (`NestedDeserialization`, `@InternalApi`), checked in two places: `Serialization.deserializeByteArray`, and `WrappedPayloadSupport.deserializePayload` — the latter calls the serializer directly for two of its three branches and so does not pass through the former. Nesting deeper than `pekko.actor.serialization-max-nesting-depth` (default 32) is rejected with a `NotSerializableException`. Ordinary nesting is unaffected; the default sits well above anything the wrapper types produce in practice. ### Result An over-nested payload is reported as an ordinary serialization failure instead of a `StackOverflowError`. No behaviour change for messages within the depth limit. ### Tests - `sbt "remote/testOnly org.apache.pekko.remote.serialization.NestedPayloadDepthSpec"` - 5 passed - `sbt "actor-tests/testOnly org.apache.pekko.serialization.SerializeSpec org.apache.pekko.serialization.WireManifestClassLoadingSpec"` - passed unchanged - `sbt "actor/mimaReportBinaryIssues" "remote/mimaReportBinaryIssues"` - no issues - `scalafmt` on the changed Scala sources ### References None - robustness of nested payload deserialization -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
