lizining1231 opened a new pull request, #3811: URL: https://github.com/apache/dubbo-go/pull/3811
### Description Fixes #3810 Related: #3747 #3497 Dubbo-go already supports RPC encoding and decoding, but introducing an external format through high-level configuration requires changes to protocol-specific selection logic. This PR adds public serialization and remoting codec extension points. Applications register a factory and select its name through configuration. Serialization controls message bodies; the remoting codec controls complete Classic Dubbo frames. Existing defaults, wire formats and legacy registration APIs remain compatible. ### Changes #### 1. Public contracts and registration - Add the public `serialization.Codec` interface and optional append, argument-type resolution and stream capabilities. - Add protocol-scoped serialization factories and named remoting codec factories through `common/extension`. - Validate names, modes, wire identities, Dubbo serialization IDs and factory results while preserving legacy registration APIs. #### 2. Configuration and receive policies - Expose serialization selection, accepted serialization lists and Dubbo framing selection through client/server options and YAML configuration. - Preserve consumer/provider defaults and reference/service overrides, including the distinction between an omitted receive list and an explicit empty list. - Build instances from local configuration snapshots; discovery and reconnection cannot replace local plugin selection or broaden receive permissions. #### 3. Triple integration - Resolve registered message and non-IDL object codecs on both gRPC wire and Triple wire. - Validate request and response formats before decoding, including empty unary and streaming messages. - Preserve declared argument types, wrapper compatibility and optional append support. - Bind codec resources and receive policies to each group/version, retaining captured contexts for active requests when handlers are replaced. #### 4. Classic Dubbo integration - Select stream serializers by the validated serialization ID in the header and support named factories for complete request/response framing. - Decode business arguments against method signatures from the original bytes, preserving wide integers, collections and structs while rejecting invalid numeric values. - Preserve baseline Hessian body behavior and legacy serializer overrides. - Isolate incompatible endpoint bindings and lease shared codec resources to Getty connections and listeners. #### 5. Independent gRPC integration - Adapt registered message codecs to stable grpc-go content subtypes and expose subtype selection through the public client. - Reject conflicting process-wide configurations or external subtype registrations. - Preserve standard Protobuf traffic and existing JSON behavior. #### 6. Cleanup, recovery and documentation - Release partially constructed codecs and invokers after client initialization, service export or cluster assembly failures. - Retain shared resources until their last user finishes, including active calls and streams; reconstruct fresh generations after final release. - Add plugin documentation covering registration, configuration, protocol constraints and lifecycle requirements, with navigation from both root READMEs. Fory is outside this PR. Custom Go plugins do not establish Java interoperability, and no performance improvement is claimed. ### Test The following lists every new top-level test function in this change: **61 root-module tests and 9 independent-module RPC scenarios**. Table-driven subtests are included under their parent functions. Two existing tests updated by this PR are listed separately. #### 1. Registration, configuration snapshots and resources | Test | Description | | --- | --- | | `TestSerializationConstruction` | Construct a codec from its registration and validate the resulting instance. | | `TestSerializationValidation` | Reject invalid serialization registrations and incompatible capabilities. | | `TestInitializeSerializationSnapshots` | Initialize extension configuration and produce independent runtime snapshots. | | `TestLocalSerializationPolicySurvivesDiscovery` | Preserve local receive policy when discovery parameters are merged. | | `TestCodecRegistrationValidationAndAtomicID` | Validate registration fields and reserve Dubbo IDs atomically. | | `TestCodecSerializationFactoryCleanup` | Release factory results when construction or validation fails. | | `TestCodecConfigurationFinalValuesAndAtomicPublication` | Publish snapshots only after successful initialization using final configuration values. | | `TestResourcesDrainAndCloseOnce` | Wait for resource users to finish and close each owned resource once. | | `TestCodecResourcesCloseErrors` | Handle resource-close failures without skipping remaining cleanup. | #### 2. Client, server and YAML configuration | Test | Description | | --- | --- | | `TestCodecDialConfigurationIsolation` | Keep Dial and DialWithDefinition settings isolated, including inherited and explicitly empty receive lists. | | `TestCodecReferenceRollback` | Release prepared reference resources after partial initialization failures. | | `TestCodecLegacyReferenceInitialization` | Preserve reference initialization for legacy protocol implementations. | | `TestCodecClusterAssemblyRollback` | Destroy created invokers when cluster lookup or assembly fails. | | `TestCodecSerializationDefaultsFollowFinalProtocol` | Resolve serialization defaults from the final selected protocol without overriding explicit choices. | | `TestAcceptedSerializationsClonePresence` | Clone receive lists without losing nil versus explicit-empty semantics. | | `TestCodecReceivePolicyYAMLAndNestedClones` | Decode YAML receive policies and keep nested configuration clones independent. | | `TestCodecServiceRollback` | Release service resources after partial export failures. | | `TestCodecServiceReceivePolicyInheritance` | Inherit provider receive policy without sharing mutable slices; retain explicit-empty overrides. | #### 3. Triple initialization, providers and invocation metadata | Test | Description | | --- | --- | | `TestCodecClientInitializationRollback` | Release codec instances when Triple client construction fails. | | `TestDiscoveredProvidersShareCodecLease` | Keep shared codec resources alive while another discovered provider still uses them. | | `TestProviderCodecConcurrentRelease` | Release provider leases safely under concurrent teardown. | | `TestCodecTripleGenerationInterleaveAndRetry` | Keep codec generations independent across release, reconstruction and retry. | | `TestCodecServerStopOwnsAllServiceCodecs` | Release service codec resources when the Triple server stops. | | `TestCodecInvocationDeclaredTypesReachWrapper` | Carry invocation-declared argument types into the Triple request wrapper. | #### 4. Triple encoding, dispatch and stream lifecycle | Test | Description | | --- | --- | | `TestConfiguredSerializationSessions` | Dispatch configured request/response codecs through serialization sessions. | | `TestDisabledProtobufEmptyEnvelope` | Reject empty Protobuf envelopes when that format is disabled. | | `TestCodecDeclaredTypesAndResolver` | Honor declared parameter types and optional plugin type resolution. | | `TestCodecRuntimeMapIsolationAndRejection` | Keep runtime codec maps isolated and reject unsupported or disabled entries. | | `TestCodecResponseSchemasAndPayloadErrors` | Validate supported response wrapper schemas and propagate malformed-payload errors. | | `TestCodecMessageSessionAppender` | Use optional append support in configured message sessions. | | `TestCodecStreamingEmptyMessagePolicy` | Apply receive-policy checks to empty streaming messages. | | `TestCodecStreamLeaseTermination` | Release stream leases on terminal operations. | | `TestCodecStreamCloseWaitsForOperations` | Keep stream resources alive until active operations finish. | | `TestCodecCompleteCallResourceBoundaries` | Retain resources across complete client and handler call boundaries. | | `TestCodecHandlerReplacementKeepsInflightContext` | Preserve captured codec contexts for requests executing during handler replacement. | #### 5. Dubbo serialization, framing and method argument decoding | Test | Description | | --- | --- | | `TestConfiguredDubboIDAndFraming` | Use the configured serialization ID and framing implementation. | | `TestDubboInitializationAndReceivePolicy` | Validate initialization settings and restrict accepted inbound serialization IDs. | | `TestCodecEndpointConflictAndBindingIsolation` | Reject incompatible shared-listener settings and isolate independent bindings. | | `TestCodecRuntimeViewAndLegacyOverride` | Preserve bound runtime views and legacy serializer overrides. | | `TestCodecConfiguredFrameBoundariesAndCorrelation` | Handle partial or adjacent frames and preserve response correlation. | | `TestStreamBaselineFixtures` | Verify stream encoding and decoding against baseline body fixtures. | | `TestIndependentJSONStreamRequest` | Decode Dubbo request fields with an independent JSON stream serializer. | | `TestIndependentJSONStreamResponse` | Decode Dubbo responses with an independent JSON stream serializer. | | `TestJSONStreamArgumentSignature` | Decode concrete integer, collection and struct arguments; reject invalid numeric conversions. | | `TestCodecStreamFailureCleanup` | Close stream resources on encoding and decoding failures. | | `TestCodecStreamSignatureRouting` | Resolve concrete method argument targets while preserving dynamic invocation behavior. | #### 6. Remoting factories and Getty ownership | Test | Description | | --- | --- | | `TestCodecFactoryAndLegacyOverride` | Select named factories while retaining legacy codec overrides. | | `TestSharedListenerCodecOwnership` | Close a shared listener binding only through its owner. | | `TestClientCodecLeasesAndLocalRecreation` | Share client leases and reconstruct resources from local settings after final release. | | `TestClientCodecConcurrentRelease` | Close client codec resources once under concurrent lease release. | | `TestCodecGenerationInterleaveAndRetry` | Keep old and new remoting codec generations independent across retries. | | `TestCodecRemotingFactoryResults` | Validate remoting factory results and clean up failed constructions. | | `TestClientCloseReleasesOnlyItsCodecLease` | Closing one Getty client must not invalidate another client sharing the codec. | | `TestCodecGettyRetryAndPublicationLeases` | Release leases after connection failure or closure before connection publication. | #### 7. Hessian stream compatibility | Test | Description | | --- | --- | | `TestStreamOwnershipAndRecovery` | Preserve stream byte ownership and recover after failed operations. | | `TestStreamReferenceScope` | Keep object reference state scoped to one complete message body. | | `TestCodecHessianAttachmentTargetsAndNull` | Decode attachment targets and null values through the Hessian stream adapter. | #### 8. Independent gRPC | Test | Description | | --- | --- | | `TestGRPCSerializationRPCAndConflict` | Complete custom-subtype RPC and reject conflicting configurations. | | `TestLegacyGRPCJSONSemantics` | Preserve existing independent gRPC JSON behavior. | | `TestCodecGRPCConflictsAndRetry` | Handle subtype registration conflicts and retry after failed construction. | | `TestCodecPublicGRPCClientDefaultSubtype` | Apply configured subtype selection through the public gRPC client. | #### 9. Independent-module RPC scenarios | Test | Description | | --- | --- | | `TestHighLevelPluginRPC` | Select external plugins through public APIs on both Triple wires and exercise all four encoding/decoding steps. | | `TestUnknownAndDisabledPlugin` | Reject unknown or disabled external plugin formats. | | `TestDubboPluginRPC` | Exercise custom Dubbo stream serialization and remoting framing over real RPC connections. | | `TestBuiltinsAndCorruptPayload` | Preserve built-in formats and reject corrupted custom payloads. | | `TestCodecSharedTripleHandlerContexts` | Verify group/version-specific codec and receive policies on a shared Triple endpoint. | | `TestCodecTripleFailedExportRetryRPC` | Complete real RPC after a failed multi-protocol export and retry. | | `TestCodecClusterAssemblyFailureReleasesLease` | Verify real Dial failures release codec leases before returning. | | `TestPureJSONStreamSharedEndpoint` | Exercise an independent JSON stream serializer on a shared Dubbo endpoint. | | `TestDubboDiscoveredProviderCodecLeases` | Keep surviving Dubbo connections usable after sibling closure or provider initialization failure. | #### 10. Updated existing tests | Test | Update | | --- | --- | | `TestCloneConfig` | Include accepted serialization lists in existing configuration clone cases. | | `Test_newClientManager_Serialization` | Expect an initialization error for unsupported formats instead of a panic. | ### Validation - Relevant root-package regression tests passed across serialization, remoting, protocol adapters, configuration and lifecycle handling. - The independent external-plugin module passed its functional RPC scenarios. - Focused race checks passed, including external Triple handler isolation, replacement and failed-export recovery scenarios. - Documentation examples compiled; YAML, links and `git diff --check` passed. The complete external-module race run still exposes an existing Getty shutdown race that also reproduces on the unmodified baseline; it is not reported as passing. Java peer interoperability was not exercised. ### Checklist - [x] I confirm the target branch is `develop` - [x] I have run `make fmt` to format my code - [x] I have run `make test` to run local tests - [x] I have added tests that prove my fix is effective or that my feature works -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
