I did some further investigation on this topic. My goal is to decouple Pulsar's protobuf and grpc versions from Bookkeeper protobuf and grpc versions.
Pulsar uses these dependencies from Bookkeeper: With <groupId>org.apache.bookkeeper</groupId>: <artifactId>bookkeeper-common-allocator</artifactId> <artifactId>bookkeeper-common</artifactId> <artifactId>bookkeeper-server</artifactId> <artifactId>bookkeeper-stats-api</artifactId> <artifactId>bookkeeper-tools-framework</artifactId> <artifactId>circe-checksum</artifactId> <artifactId>codahale-metrics-provider</artifactId> <artifactId>cpu-affinity</artifactId> <artifactId>datasketches-metrics-provider</artifactId> <artifactId>prometheus-metrics-provider</artifactId> <artifactId>stream-storage-java-client</artifactId> <artifactId>stream-storage-server</artifactId> <artifactId>stream-storage-tests-common</artifactId> <artifactId>vertx-http-server</artifactId> With <groupId>org.apache.distributedlog</groupId>: <artifactId>distributedlog-core</artifactId> It turns out that there's already a shaded dependency bookkeeper-server-shaded that shades protobuf-java . However, the problem is that switching to the shaded jar wouldn't help alone since the distributedlog-core and stream-storage-server aren't shaded and there are remaining unshaded protobuf and grpc dependencies that would get pulled in. I noticed that a distributedlog-core-shaded also exists. That would solve some problems but not all. The stream-storage-server and stream-storage-java-client would still pull in unshaded dependencies. Publishing shaded versions of those dependencies would add a lot of duplication and the end result not be great since the Pulsar distribution size would increase. One possible solution would be to shade grpc and protobuf dependencies within bookkeeper and publish them as libraries which the other bookkeeper modules would depend on. One of the challenges in this approach is that there would have to be a way to get the protoc generated protobuf and grpc stub code to use the relocated packages. I guess a search&replace for the generated code could handle that, would have to investigate if this approach is used in other OSS projects to solve the same problem. Do others have suggestions about the approach to take for shading protobuf & grpc across all dependencies? In Pulsar, we need to be able to upgrade GRPC so that we can resolve CVE-2023-32732. It's not that this CVE would impact Pulsar. In case of CVE-2023-32732, it's just about upgrading the library so that Pulsar users don't need to worry about the possible vulnerability impact. Many enterprises have policies where every CVE in the vulnerability report needs to be resolved. When we upgrade GRPC, we have one less CVEs on the list. It seems that for existing Bookkeeper release branches, we would need to coordinate the protobuf & grpc library version upgrades in the same way as we have done in the past since the change to shade the protobuf & grpc versions is not a small change that we could easily backport to older versions. Do others agree about this? If so, should we upgrade protobuf & grpc in branch-4.16 ? how about branch-4.15 ? -Lari On 2023/12/21 02:58:12 Yong Zhang wrote: > Looks like shading the dependency is a good idea. It can break the > dependency cycle. > > +1 to shade the dependency > > Best regards, > Yong > > On Fri, 15 Dec 2023 at 02:58, Lari Hotari <lhot...@apache.org> wrote: > > > I would like to make a minor correction to my previous email: > > > > The pull request https://github.com/apache/bookkeeper/pull/3992 has been > > merged into the master branch and not rolled back. Consequently, > > CVE-2023-32732 has been resolved in the master branch with gRPC 1.56.0. > > However, this change was only rolled back in the branch-4.16 due to the > > compatibility issues described in the previous email. > > > > Looking ahead, once version 4.17 is released from the master branch, we > > should be able to use it in Pulsar. This will enable us to upgrade gRPC and > > protobuf to match the versions used in Bookkeeper, as we have done > > previously. However, it would be great if we could find a solution to break > > this dependency cycle. > > > > -Lari > > > > On 2023/12/14 18:42:04 Lari Hotari wrote: > > > Dear all, > > > > > > I'm reaching out to discuss an ongoing issue in Pulsar related to > > CVE-2023-32732, which necessitates upgrading gRPC in Pulsar. Although this > > CVE isn't critical, it's flagged by CVE scanners, and addressing it > > requires careful coordination of upgrades for gRPC and Protobuf libraries > > in both Pulsar and Bookkeeper. > > > > > > Currently, we face a challenge due to a strong dependency on gRPC and > > Protobuf library versions: upgrades must first occur in Bookkeeper and then > > in Pulsar to maintain compatibility. > > > On the Pulsar side, the primary issues stem from Protobuf version > > upgrades in addition to the gRPC incompatibility with the Bookkeeper > > StateStore client. > > > > > > In June I made an attempt to upgrade grpc and protobuf in both Pulsar > > and Bookkeeper: > > > https://github.com/apache/pulsar/pull/20602 (this was never merged due > > to failing tests) > > > https://github.com/apache/bookkeeper/pull/3992 (this was merged, but it > > had to be roll backed) > > > https://github.com/apache/bookkeeper/pull/4000 (closed as invalid > > solution) > > > > > > In Bookkeeper, we had cherry-picked the upgrade to branch-4.16 and that > > had to be downgraded > > > with https://github.com/apache/bookkeeper/pull/4001, because of > > compatibility issues. > > > > > > A discovery during these changes was the compatibility policy of > > protobuf (https://protobuf.dev/support/cross-version-runtime-guarantee/), > > which states that cross-version runtime support isn't guaranteed. I have > > also understood that grpc has a similar policy. This policy poses a > > challenge, as it necessitates the use of identical grpc and protobuf > > versions across both Pulsar and Bookkeeper. > > > > > > To address this, one potential solution could be shading grpc & protobuf > > in the bookkeeper client that exposes these dependencies. This might > > resolve the compatibility issue in the bookkeeper client libraries. > > > > > > I welcome your thoughts and suggestions on this matter. > > > > > > Best regards, > > > > > > Lari > > > > > >
