[
https://issues.apache.org/jira/browse/BEAM-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17293819#comment-17293819
]
Tomo Suzuki edited comment on BEAM-11227 at 3/10/21, 8:36 PM:
--------------------------------------------------------------
Looking at Boury Mbodj's
[activity|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=bmbodj&selectedTab=com.atlassian.streams.streams-jira-plugin:user-profile-stream-panel],
it seems that this is one-off ticket (not by automation).
[~kenn] Sure. Let me continue https://github.com/apache/beam/pull/14028 to see
what would break.
h1. Memo
Previous emails
* [[VOTE] Vendored Dependencies Release gRPC 1.26.0 v0.3 for BEAM-9288 RC
#3|https://lists.apache.org/thread.html/rea4a27c47529a27936ab2c51162c8e532b8b625c4d70c4f7f485c7cd%40%3Cdev.beam.apache.org%3E]
(vote passed)
* [[VOTE] Vendored Dependencies Release gRPC 1.26.0 v0.3 for BEAM-9288 RC
#2|https://lists.apache.org/thread.html/rc6372c9873a2b00cf5dc30efeeb0b13bb1aa92a0f93e2417211effc4%40%3Cdev.beam.apache.org%3E]
(commit id mistake)
* [Re: [VOTE] Vendored Dependencies Release gRPC 1.26.0 v0.3 for
BEAM-9288|https://lists.apache.org/thread.html/r31fb38e8480889ecb23db7135771d419c9cf43fd20be96c4aa179e54%40%3Cdev.beam.apache.org%3E]
(-> conscrypt was in JAR)
{code}
suztomo@suztomo:~/beam$ jar tvf
vendor/grpc-1_36_0/build/libs/beam-vendor-grpc-1_36_0-0.1.jar | grep conscrypt
suztomo@suztomo:~/beam$
{code}
Previous version (beam-vendor-grpc-1_26_0-0.3):
{code}
suztomo-macbookpro44% jar tvf ~/Downloads/beam-vendor-grpc-1_26_0-0.3.jar|grep
'\.so$'
2626449 Sat Sep 21 10:06:14 EDT 2019
META-INF/native/liborg_apache_beam_vendor_grpc_v1p26p0_netty_tcnative_linux_x86_64.so
59545 Wed Nov 26 20:02:18 EST 2014 linux/amd64/liblz4-java.so
68840 Wed Nov 26 20:02:18 EST 2014 linux/i386/liblz4-java.so
161360 Wed Nov 26 20:02:18 EST 2014 win32/amd64/liblz4-java.so
{code}
Current proposal
{code}
suztomo@suztomo:~/beam$ jar tvf
vendor/grpc-1_36_0/build/libs/beam-vendor-grpc-1_36_0-0.1.jar | grep '\.so$'
2628280 Fri Aug 21 11:19:38 UTC 2020
META-INF/native/liborg_apache_beam_vendor_grpc_v1p36p0_netty_tcnative_linux_x86_64.so
1933284 Fri Aug 21 11:19:38 UTC 2020
META-INF/native/liborg_apache_beam_vendor_grpc_v1p36p0_netty_tcnative_linux_aarch64.so
59545 Wed Nov 26 20:02:18 UTC 2014 linux/amd64/liblz4-java.so
68840 Wed Nov 26 20:02:18 UTC 2014 linux/i386/liblz4-java.so
161360 Wed Nov 26 20:02:18 UTC 2014 win32/amd64/liblz4-java.so
{code}
h1. Consideration
h2. protobuf-java version
gRPC 1.36 uses protobuf-java 3.12.0 and
com.google.api.grpc:proto-google-common-protos:2.0.1 draws protobuf-java
3.13.0. Gradle's dependency mediation chooses 3.13.0 (higher) which has the
Java8-incompatibility problem
(https://github.com/protocolbuffers/protobuf/issues/7827). We need to set a
higher version to avoid having the problem in the vendored gRPC.
h2. Remove dependencies that seem unused
In preparing the vendored gRPC 1.26, it seems that unnecessary dependencies
were added to the vendored gRPC project just to resolve the error messages by
Linkage Checker. Let's verify whether they are really needed or not.
{code}
"io.perfmark:perfmark-api:$perfmark_version",
"com.github.jponge:lzma-java:$lzma_java_version",
"com.google.protobuf.nano:protobuf-javanano:$protobuf_javanano_version",
"com.jcraft:jzlib:$jzlib_version",
"com.ning:compress-lzf:$compress_lzf_version",
"net.jpountz.lz4:lz4:$lz4_version",
"org.bouncycastle:bcpkix-jdk15on:$bouncycastle_version",
"org.bouncycastle:bcprov-jdk15on:$bouncycastle_version",
"org.eclipse.jetty.alpn:alpn-api:$alpn_api_version",
"org.eclipse.jetty.npn:npn-api:$npn_api_version",
"org.jboss.marshalling:jboss-marshalling:$jboss_marshalling_version",
"org.jboss.modules:jboss-modules:$jboss_modules_version"
{code}
If they are not needed by Beam's use of gRPC, then we can
* remove the source class that appear in the linkage errors
* add [exclusion
file|https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/Linkage-Checker-Exclusion-File]
for Linkage Checker
Linkage Errors when I remove them:
https://gist.github.com/suztomo/6e76e09b33ff834726634175c458f096
(continuing)
h1. Linkage Errors
I ran checkJavaLinkage task (
[output|https://gist.github.com/suztomo/c1d9c587ea58ad4cfd25c853ad921c93#file-beam-vendor-grpc-1_36_0_with_protobuf_3-15-txt-L6].
Apart from the errors on loggers, it shows the following linkage errors on
{{io.netty.handler.ssl.ConscryptAlpnSslEngine}}:
{code}
Class org.junit.runners.model.Statement is not found;
referenced by 1 class file
org.apache.beam.vendor.grpc.v1p36p0.io.grpc.testing.GrpcCleanupRule
(org.apache.beam:beam-vendor-grpc-1_36_0:0.1)
Cause:
Unknown
Class org.junit.runners.model.MultipleFailureException is not found;
referenced by 1 class file
org.apache.beam.vendor.grpc.v1p36p0.io.grpc.testing.GrpcCleanupRule
(org.apache.beam:beam-vendor-grpc-1_36_0:0.1)
Cause:
Unknown
Class org.junit.rules.ExternalResource is not found;
referenced by 1 class file
org.apache.beam.vendor.grpc.v1p36p0.io.grpc.testing.GrpcServerRule
(org.apache.beam:beam-vendor-grpc-1_36_0:0.1)
Cause:
Unknown
Class org.junit.rules.TestRule is not found;
referenced by 1 class file
org.apache.beam.vendor.grpc.v1p36p0.io.grpc.testing.GrpcCleanupRule
(org.apache.beam:beam-vendor-grpc-1_36_0:0.1)
Cause:
Unknown
Class org.conscrypt.AllocatedBuffer is not found;
referenced by 1 class file
org.apache.beam.vendor.grpc.v1p36p0.io.netty.handler.ssl.ConscryptAlpnSslEngine
(org.apache.beam:beam-vendor-grpc-1_36_0:0.1)
Cause:
Unknown
Class org.conscrypt.Conscrypt is not found;
referenced by 1 class file
org.apache.beam.vendor.grpc.v1p36p0.io.netty.handler.ssl.ConscryptAlpnSslEngine
(org.apache.beam:beam-vendor-grpc-1_36_0:0.1)
Cause:
Unknown
Class org.conscrypt.BufferAllocator is not found;
referenced by 1 class file
org.apache.beam.vendor.grpc.v1p36p0.io.netty.handler.ssl.ConscryptAlpnSslEngine
(org.apache.beam:beam-vendor-grpc-1_36_0:0.1)
Cause:
Unknown
Class org.conscrypt.HandshakeListener is not found;
referenced by 1 class file
org.apache.beam.vendor.grpc.v1p36p0.io.netty.handler.ssl.ConscryptAlpnSslEngine
(org.apache.beam:beam-vendor-grpc-1_36_0:0.1)
Cause:
Unknown
{code}
The {{grpc.testing}} package seems fine. How about the
{{org.conscrypt.BufferAllocator}}? Are they safe?
* Why aren't they part of the vendored JAR in version 1.36?
How it it declared?
* Were the class files part of the vendored gRPC 1.26?
* Does Beam use it?
was (Author: suztomo):
Looking at Boury Mbodj's
[activity|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=bmbodj&selectedTab=com.atlassian.streams.streams-jira-plugin:user-profile-stream-panel],
it seems that this is one-off ticket (not by automation).
[~kenn] Sure. Let me continue https://github.com/apache/beam/pull/14028 to see
what would break.
h1. Memo
Previous emails
* [[VOTE] Vendored Dependencies Release gRPC 1.26.0 v0.3 for BEAM-9288 RC
#3|https://lists.apache.org/thread.html/rea4a27c47529a27936ab2c51162c8e532b8b625c4d70c4f7f485c7cd%40%3Cdev.beam.apache.org%3E]
(vote passed)
* [[VOTE] Vendored Dependencies Release gRPC 1.26.0 v0.3 for BEAM-9288 RC
#2|https://lists.apache.org/thread.html/rc6372c9873a2b00cf5dc30efeeb0b13bb1aa92a0f93e2417211effc4%40%3Cdev.beam.apache.org%3E]
(commit id mistake)
* [Re: [VOTE] Vendored Dependencies Release gRPC 1.26.0 v0.3 for
BEAM-9288|https://lists.apache.org/thread.html/r31fb38e8480889ecb23db7135771d419c9cf43fd20be96c4aa179e54%40%3Cdev.beam.apache.org%3E]
(-> conscrypt was in JAR)
{code}
suztomo@suztomo:~/beam$ jar tvf
vendor/grpc-1_36_0/build/libs/beam-vendor-grpc-1_36_0-0.1.jar | grep conscrypt
suztomo@suztomo:~/beam$
{code}
Previous version (beam-vendor-grpc-1_26_0-0.3):
{code}
suztomo-macbookpro44% jar tvf ~/Downloads/beam-vendor-grpc-1_26_0-0.3.jar|grep
'\.so$'
2626449 Sat Sep 21 10:06:14 EDT 2019
META-INF/native/liborg_apache_beam_vendor_grpc_v1p26p0_netty_tcnative_linux_x86_64.so
59545 Wed Nov 26 20:02:18 EST 2014 linux/amd64/liblz4-java.so
68840 Wed Nov 26 20:02:18 EST 2014 linux/i386/liblz4-java.so
161360 Wed Nov 26 20:02:18 EST 2014 win32/amd64/liblz4-java.so
{code}
Current proposal
{code}
suztomo@suztomo:~/beam$ jar tvf
vendor/grpc-1_36_0/build/libs/beam-vendor-grpc-1_36_0-0.1.jar | grep '\.so$'
2628280 Fri Aug 21 11:19:38 UTC 2020
META-INF/native/liborg_apache_beam_vendor_grpc_v1p36p0_netty_tcnative_linux_x86_64.so
1933284 Fri Aug 21 11:19:38 UTC 2020
META-INF/native/liborg_apache_beam_vendor_grpc_v1p36p0_netty_tcnative_linux_aarch64.so
59545 Wed Nov 26 20:02:18 UTC 2014 linux/amd64/liblz4-java.so
68840 Wed Nov 26 20:02:18 UTC 2014 linux/i386/liblz4-java.so
161360 Wed Nov 26 20:02:18 UTC 2014 win32/amd64/liblz4-java.so
{code}
h1. Consideration
h2. protobuf-java version
gRPC 1.36 uses protobuf-java 3.12.0 and
com.google.api.grpc:proto-google-common-protos:2.0.1 draws protobuf-java
3.13.0. Gradle's dependency mediation chooses 3.13.0 (higher) which has the
Java8-incompatibility problem
(https://github.com/protocolbuffers/protobuf/issues/7827). We need to set a
higher version to avoid having the problem in the vendored gRPC.
h1. Remove dependencies that seem unused
In preparing the vendored gRPC 1.26, it seems that unnecessary dependencies
were added to the vendored gRPC project just to resolve the error messages by
Linkage Checker. Let's verify whether they are really needed or not.
{code}
"io.perfmark:perfmark-api:$perfmark_version",
"com.github.jponge:lzma-java:$lzma_java_version",
"com.google.protobuf.nano:protobuf-javanano:$protobuf_javanano_version",
"com.jcraft:jzlib:$jzlib_version",
"com.ning:compress-lzf:$compress_lzf_version",
"net.jpountz.lz4:lz4:$lz4_version",
"org.bouncycastle:bcpkix-jdk15on:$bouncycastle_version",
"org.bouncycastle:bcprov-jdk15on:$bouncycastle_version",
"org.eclipse.jetty.alpn:alpn-api:$alpn_api_version",
"org.eclipse.jetty.npn:npn-api:$npn_api_version",
"org.jboss.marshalling:jboss-marshalling:$jboss_marshalling_version",
"org.jboss.modules:jboss-modules:$jboss_modules_version"
{code}
If they are not needed by Beam's use of gRPC, then we can
* remove the source class that appear in the linkage errors
* add [exclusion
file|https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/Linkage-Checker-Exclusion-File]
for Linkage Checker
h1. Linkage Errors
I ran checkJavaLinkage task (
[output|https://gist.github.com/suztomo/c1d9c587ea58ad4cfd25c853ad921c93#file-beam-vendor-grpc-1_36_0_with_protobuf_3-15-txt-L6].
Apart from the errors on loggers, it shows the following linkage errors on
{{io.netty.handler.ssl.ConscryptAlpnSslEngine}}:
{code}
Class org.junit.runners.model.Statement is not found;
referenced by 1 class file
org.apache.beam.vendor.grpc.v1p36p0.io.grpc.testing.GrpcCleanupRule
(org.apache.beam:beam-vendor-grpc-1_36_0:0.1)
Cause:
Unknown
Class org.junit.runners.model.MultipleFailureException is not found;
referenced by 1 class file
org.apache.beam.vendor.grpc.v1p36p0.io.grpc.testing.GrpcCleanupRule
(org.apache.beam:beam-vendor-grpc-1_36_0:0.1)
Cause:
Unknown
Class org.junit.rules.ExternalResource is not found;
referenced by 1 class file
org.apache.beam.vendor.grpc.v1p36p0.io.grpc.testing.GrpcServerRule
(org.apache.beam:beam-vendor-grpc-1_36_0:0.1)
Cause:
Unknown
Class org.junit.rules.TestRule is not found;
referenced by 1 class file
org.apache.beam.vendor.grpc.v1p36p0.io.grpc.testing.GrpcCleanupRule
(org.apache.beam:beam-vendor-grpc-1_36_0:0.1)
Cause:
Unknown
Class org.conscrypt.AllocatedBuffer is not found;
referenced by 1 class file
org.apache.beam.vendor.grpc.v1p36p0.io.netty.handler.ssl.ConscryptAlpnSslEngine
(org.apache.beam:beam-vendor-grpc-1_36_0:0.1)
Cause:
Unknown
Class org.conscrypt.Conscrypt is not found;
referenced by 1 class file
org.apache.beam.vendor.grpc.v1p36p0.io.netty.handler.ssl.ConscryptAlpnSslEngine
(org.apache.beam:beam-vendor-grpc-1_36_0:0.1)
Cause:
Unknown
Class org.conscrypt.BufferAllocator is not found;
referenced by 1 class file
org.apache.beam.vendor.grpc.v1p36p0.io.netty.handler.ssl.ConscryptAlpnSslEngine
(org.apache.beam:beam-vendor-grpc-1_36_0:0.1)
Cause:
Unknown
Class org.conscrypt.HandshakeListener is not found;
referenced by 1 class file
org.apache.beam.vendor.grpc.v1p36p0.io.netty.handler.ssl.ConscryptAlpnSslEngine
(org.apache.beam:beam-vendor-grpc-1_36_0:0.1)
Cause:
Unknown
{code}
The {{grpc.testing}} package seems fine. How about the
{{org.conscrypt.BufferAllocator}}? Are they safe?
* Why aren't they part of the vendored JAR in version 1.36?
How it it declared?
* Were the class files part of the vendored gRPC 1.26?
* Does Beam use it?
> Upgrade beam-vendor-grpc-1_26_0-0.3 to fix CVE-2020-27216
> ---------------------------------------------------------
>
> Key: BEAM-11227
> URL: https://issues.apache.org/jira/browse/BEAM-11227
> Project: Beam
> Issue Type: Bug
> Components: build-system
> Affects Versions: 2.21.0, 2.22.0, 2.23.0, 2.24.0, 2.25.0
> Reporter: Boury Mbodj
> Priority: P1
> Labels: apache-beam, beam
> Fix For: 2.29.0
>
> Time Spent: 21h 20m
> Remaining Estimate: 0h
>
> *+Description+**:* [Apache Beam :: Vendored Dependencies :: GRPC ::
> 1.26.0|https://mvnrepository.com/artifact/org.apache.beam/beam-vendor-grpc-1_26_0]
> »
> [0.3|https://mvnrepository.com/artifact/org.apache.beam/beam-vendor-grpc-1_26_0/0.3]
> uses the dependency Eclipse Jetty (9.2.10.v20150310), which is prone to a
> privilege escalation vulnerability. This issue (CVE-2020-27216) was published
> on 23/10/2020.
> *+Affected Versions:+*
> Eclipse Jetty versions 9.4.32.v20200930 and prior, 10.0.0.beta2 and prior
> and 11.0.0.beta2 and prior.
> *+Recommendation/+* *+Update Suggestion:+*
> Update the Eclipse Jetty dependency to version 9.4.33.v20201020,
> 10.0.0.beta3, 11.0.0.beta3 or later.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
