[
https://issues.apache.org/jira/browse/HDDS-15593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Cheng Li updated HDDS-15593:
--------------------------------
Description:
Fix protobuf/gRPC generated-source rewriting in {{hdds-interface-client}} and
{{{}hdds-interface-server{}}}.
These modules generate Ratis protocol classes that must use Ratis-shaded
packages, for example:
org.apache.ratis.thirdparty.io.grpcorg.apache.ratis.thirdparty.com.google.protobuforg.apache.ratis.thirdparty.com.google.common
The previous Ant {{<replace dir="...">}} configuration did not reliably rewrite
nested generated Java files under:
{code:java}
target/generated-sources/proto-java-for-ratis/org/apache/hadoop/... {code}
This patch changes the replacement rules to use explicit recursive
{{<fileset>}} entries over {{{}**/*.java{}}}.
h3. Reproduction
A targeted clean build can fail before this patch:
mvn clean
mvn clean test-compile -pl hadoop-ozone/s3gateway -am
Example error:
XceiverClientProtocolServiceGrpc.java:[3,22] package io.grpc does not exist
IntraDatanodeProtocolServiceGrpc.java:[7,26] package io.grpc.stub.annotations
does not exist{{}}
This may be hidden by a prior full build or local cache, because downstream
module-only tests can resolve already-installed snapshot artifacts from
{{{}~/.m2{}}}.
was:
`hdds-interface-client` and `hdds-interface-server` generate protobuf/gRPC Java
sources that are expected to use Ratis-shaded dependencies:
- `org.apache.ratis.thirdparty.io.grpc`
- `org.apache.ratis.thirdparty.com.google.protobuf`
- `org.apache.ratis.thirdparty.com.google.common`
The existing `maven-antrun-plugin` replacement used `<replace dir="...">`,
which did not reliably apply to generated Java files under nested package
directories such as:
```text
target/generated-sources/proto-java-for-ratis/org/apache/hadoop/...
```
As a result, clean builds could leave generated gRPC sources importing unshaded
packages like `io.grpc`, causing interface module compilation failures and
downstream missing-proto-class errors.
This patch changes the replacement rules to use explicit recursive Ant
`<fileset>` entries over `*{*}/{*}.java`.
No protobuf definitions or generated sources are checked in by this change.
Why
A clean build can fail in `hdds-interface-client` or `hdds-interface-server`
because generated Ratis protobuf/gRPC sources still reference unshaded packages.
This can also surface downstream as missing generated classes such as:
- `ContainerProtos`
- `HddsProtos`
- `XceiverClientProtocolServiceGrpc`
Reproduction before the fix
On a clean checkout without this patch:
```bash
mvn -pl :hdds-interface-client clean compile \
-DskipTests -DskipShade -DskipRecon -DskipDocs
```
Example failure:
```text
target/generated-sources/proto-java-for-ratis/.../XceiverClientProtocolServiceGrpc.java:[3,22]
package io.grpc does not exist
target/generated-sources/proto-java-for-ratis/.../IntraDatanodeProtocolServiceGrpc.java:[7,26]
package io.grpc.stub.annotations does not exist
```
A downstream build can also fail during test compilation, for example:
```bash
mvn clean test-compile -pl hadoop-ozone/s3gateway -am
```
with errors like:
```text
package org.apache.hadoop.hdds.protocol.datanode.proto does not exist
cannot find symbol: class HddsProtos
cannot find symbol: class ContainerCommandRequestProto
```
> Fix recursive Ratis protobuf import rewriting
> ---------------------------------------------
>
> Key: HDDS-15593
> URL: https://issues.apache.org/jira/browse/HDDS-15593
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Chu Cheng Li
> Assignee: Chu Cheng Li
> Priority: Major
> Labels: pull-request-available
>
> Fix protobuf/gRPC generated-source rewriting in {{hdds-interface-client}} and
> {{{}hdds-interface-server{}}}.
> These modules generate Ratis protocol classes that must use Ratis-shaded
> packages, for example:
> org.apache.ratis.thirdparty.io.grpcorg.apache.ratis.thirdparty.com.google.protobuforg.apache.ratis.thirdparty.com.google.common
>
> The previous Ant {{<replace dir="...">}} configuration did not reliably
> rewrite nested generated Java files under:
> {code:java}
> target/generated-sources/proto-java-for-ratis/org/apache/hadoop/... {code}
> This patch changes the replacement rules to use explicit recursive
> {{<fileset>}} entries over {{{}**/*.java{}}}.
> h3. Reproduction
> A targeted clean build can fail before this patch:
> mvn clean
> mvn clean test-compile -pl hadoop-ozone/s3gateway -am
>
> Example error:
> XceiverClientProtocolServiceGrpc.java:[3,22] package io.grpc does not exist
> IntraDatanodeProtocolServiceGrpc.java:[7,26] package io.grpc.stub.annotations
> does not exist{{}}
> This may be hidden by a prior full build or local cache, because downstream
> module-only tests can resolve already-installed snapshot artifacts from
> {{{}~/.m2{}}}.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
