[
https://issues.apache.org/jira/browse/HDDS-14949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy updated HDDS-14949:
----------------------------------
Epic Link: HDDS-17349
> Move OM gRPC to ratis-thirdparty shaded gRPC/Netty
> --------------------------------------------------
>
> Key: HDDS-14949
> URL: https://issues.apache.org/jira/browse/HDDS-14949
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> Every Ozone client carries two complete gRPC/Netty stacks. The datanode
> client already uses the shaded copy from \{{ratis-thirdparty}}. The OM gRPC
> transport (\{{GrpcOmTransport}}, \{{GrpcOMFailoverProxyProvider}},
> \{{GrpcOMFollowerReadFailoverProxyProvider}} in \{{ozone-common}}) is the
> only client code that still uses unshaded \{{io.grpc}} and \{{io.netty}}. So
> the shaded filesystem jar ships a second, relocated copy of both, together
> with its own BoringSSL and transport natives for 5 platforms.
> The OM gRPC transport is off by default: clients use Hadoop RPC unless
> \{{ozone.om.transport.class}} selects \{{GrpcOmTransportFactory}}. A default
> client never runs any of this code.
> Measured in \{{ozone-filesystem-hadoop3}} built from master (compressed size
> inside the jar):
> ||what||size||why it is there||
> |unshaded \{{io.grpc}}, relocated to \{{org.apache.ozone.shaded.io.grpc}}|1.3
> MB|\{{GrpcOmTransport}}|
> |unshaded \{{io.netty}} classes, relocated|4.0 MB|\{{grpc-netty}}|
> |\{{liborg_apache_ozone_shaded_netty_*}} natives: tcnative for 5 platforms,
> epoll, kqueue, resolver-dns|6.2 MB|added explicitly by
> \{{hadoop-ozone/ozonefs-shaded/pom.xml}}|
> |\{{proto-google-common-protos}}|2.7 MB|\{{grpc-protobuf}}|
> |*total*|*14.2 MB compressed, about 15 MB on disk*| |
> After HDDS-17772 removes RocksDB from \{{ozone-common}}, the filesystem jar
> is 71 MB, so this is about a fifth of it. The jar is shipped with every job
> that uses \{{-libjars}} or \{{spark.jars}}, and its size grew from 68.8 MB in
> 2.2.0 to 155 MB on master.
> The shaded stack that the datanode client already uses can carry OM gRPC as
> well. ratis-thirdparty 1.1.1 shades the same versions that Ozone pins today:
> gRPC 1.82.4 and Netty 4.1.139.Final.
> h3. Why this is not a package rename
> {\{OMRequest}} and \{{OMResponse}} are generated with unshaded
> \{{com.google.protobuf}} (\{{hadoop-ozone/interface-client}}), and Hadoop RPC
> and the OM request pipeline share them. The generated
> \{{OzoneManagerServiceGrpc}} stub builds its marshaller with
> \{{ProtoUtils.marshaller(OMRequest.getDefaultInstance())}}, and the shaded
> \{{ProtoUtils}} accepts only shaded protobuf messages. Rewriting the
> \{{io.grpc}} prefix in the generated code, as
> \{{hadoop-hdds/interface-client}} does, does not compile here. Moving the OM
> protos to shaded protobuf is out of scope.
> h3. Proposed change
> # *Binding:* a new hand-written class in \{{hadoop-ozone/common}} (e.g.
> \{{OzoneManagerServiceGrpcBinding}}). It replaces the generated stub with:
> #* a shaded \{{MethodDescriptor<OMRequest, OMResponse>}}, \{{UNARY}}, named
> \{{hadoop.ozone.OzoneManagerService/submitRequest}}, the same name as today;
> #* marshallers that serialize with \{{writeTo}} and parse with
> \{{OMRequest.parser()}} / \{{OMResponse.parser()}}, so the bytes on the wire
> don't change;
> #* a \{{ServerServiceDefinition}} for the server and a blocking-call helper
> for the client.
> # *Client:* \{{GrpcOmTransport}} and the two failover proxy providers move to
> the shaded \{{NettyChannelBuilder}}, \{{ManagedChannel}}, \{{Status}} and TLS
> classes. The interceptors and constants in \{{om/protocolPB/grpc}} move with
> them.
> # *Server:* \{{OzoneManagerServiceGrpc}} implements the binding instead of
> the generated \{{OzoneManagerServiceImplBase}}. \{{GrpcOzoneManagerServer}}
> moves to the shaded \{{NettyServerBuilder}}, and TLS uses the shaded
> \{{GrpcSslContexts}} with \{{SecurityConfig.getGrpcSslProvider()}}, as the
> datanode does. The \{{GrpcMetrics*}} interceptors in \{{hdds-framework}} move
> to the shaded \{{io.grpc}}.
> # *Build:*
> #* remove the \{{protoc-gen-grpc-java}} execution from
> \{{hadoop-ozone/interface-client}}; the message classes are generated as
> before;
> #* remove the unshaded \{{io.grpc}} / \{{io.netty}} dependencies from
> \{{ozone-common}}, \{{ozone-manager}}, \{{hdds-framework}} and
> \{{interface-client}};
> #* remove the explicit unshaded Netty native dependencies and their renames
> from \{{ozonefs-shaded}};
> #* add a check that fails the build if \{{org/apache/ozone/shaded/io/grpc}}
> or \{{liborg_apache_ozone_shaded_netty_}} reappears in the filesystem jars.
> *Out of scope.* The review of the earlier PR (#10030) asked to split these
> out:
> * new modules for the datanode and SCM protos;
> * the macOS clean and refresh workarounds in the root \{{pom.xml}};
> * the \{{ContainerStateMachine}} reordering;
> * test assertion changes;
> * switching the OM server to epoll, which is HDDS-16985.
> h3. Expected result
> ||jar||before||after||
> |\{{ozone-filesystem-hadoop3}} with HDDS-17772|71 MB|about 56 MB|
> |\{{ozone-filesystem-hadoop3}} on master today|155 MB|about 140 MB|
> {\{ozone-filesystem-hadoop2}} should shrink by about the same amount; to be
> measured. Clients that select \{{GrpcOmTransportFactory}} keep working,
> because the shaded gRPC stack stays in the jar for the datanode client.
> h3. Additional benefits
> * *Dist:* about 14.9 MB less in \{{share/ozone/lib}}: \{{grpc-*}} (8 jars),
> \{{netty-*}} (20 jars, including 5 \{{netty-tcnative-boringssl-static}}
> classifier jars), \{{proto-google-common-protos}} and \{{perfmark-api}},
> together with their \{{jar-report.txt}} and \{{LICENSE.txt}} entries. This
> applies only to the jars that \{{dependency:tree}} shows nothing else needs.
> * *One gRPC/Netty stack per JVM* in OM and S3 Gateway, which today run both.
> That means one pooled allocator, one set of default event-loop threads and
> one BoringSSL load instead of two.
> * *Epoll for the OM gRPC server (HDDS-16985) becomes a small change on top of
> this.* The shaded \{{ratis-thirdparty-misc}} has epoll natives for
> \{{x86_64}} and \{{aarch_64}}. The dist ships unshaded
> \{{netty-transport-native-epoll}} without its \{{linux-*}} classifier, so
> \{{GrpcOzoneManagerServer}} hard-codes NIO today.
> * *Fewer conflicts for applications.* Applications that depend on the
> unshaded \{{ozone-client}} (Spark, Hive, Flink, Impala) no longer get
> \{{io.grpc}} and \{{io.netty}} transitively, so they no longer clash with the
> applications' own versions.
> * *One Netty and gRPC to patch.* Security fixes come through
> ratis-thirdparty, which Ozone already has to update for the datanode path.
> Dependency scanners report one copy instead of two.
> h3. Compatibility
> * *Wire:* unchanged. The method name and the message bytes are the same, so
> an old S3 Gateway works with a new OM and the reverse, and rolling upgrades
> are not affected.
> * *Proto / RPC:* no \{{.proto}} changes. Only the generated gRPC stub class
> goes away, and only OM and \{{GrpcOmTransport}} use it.
> * *Config:* no new keys. \{{ozone.om.transport.class}} and the
> \{{ozone.om.grpc.*}} keys keep their meaning.
> h3. Risks
> * *Netty and gRPC versions follow ratis-thirdparty.* Today both match exactly
> (1.82.4 and 4.1.139.Final). From then on, a Netty or gRPC security fix for
> this path needs a ratis-thirdparty release, as it already does for the
> datanode path.
> * *\{{-Dio.netty.*}} system properties:* the shaded Netty may read relocated
> property names. Check which properties still apply before documenting any.
> * *Tests:* \{{ratis-thirdparty-misc}} has no shaded \{{grpc-inprocess}} or
> \{{grpc-testing}}. \{{TestS3GrpcOmTransport}} and
> \{{TestGrpcOmTransportConcurrentFailover}} must move from
> \{{InProcess*Builder}} / \{{GrpcCleanupRule}} to a shaded
> \{{NettyServerBuilder}} on an ephemeral localhost port. Share one marshaller
> between the tests and \{{GrpcOmTransport}}.
> * *Overlap:* HDDS-16740 and HDDS-16741 change the same \{{GrpcMetrics*}}
> interceptors. Land those first.
> h3. Testing
> * *Unit:*
> ** a marshaller round trip for \{{OMRequest}} / \{{OMResponse}}, including a
> payload close to \{{ozone.om.grpc.maximum.response.length}};
> ** the new method's full name equals
> \{{OzoneManagerServiceGrpc.getSubmitRequestMethod().getFullMethodName()}},
> checked before the stub is removed;
> ** the ported transport and failover tests.
> * *Integration:* the S3 Gateway integration tests and \{{TestS3SDK}}.
> * *Secure compose cluster with gRPC TLS:*
> ** S3 Gateway to OM traffic;
> ** an ofs client with \{{ozone.om.transport.class}} =
> \{{GrpcOmTransportFactory}};
> ** mixed versions in both directions.
> * *Jar contents:* the filesystem jars contain no
> \{{org/apache/ozone/shaded/io/grpc}}, no
> \{{org/apache/ozone/shaded/io/netty}} and no
> \{{liborg_apache_ozone_shaded_netty_*}}. Record each jar's size before and
> after.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]