[
https://issues.apache.org/jira/browse/BEAM-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16965494#comment-16965494
]
Kenneth Knowles commented on BEAM-8497:
---------------------------------------
We are aware of the problem and generally agree with your suggestion. The
problem is that upstream dependencies have Guava, etc, on their public API
surface.
When storage system XYZ ships a client library with some dependency ABC, our
default approach is assume users of storage system XYZ already have to deal
with this leaked dependency ABC. In other words, we try not to make things
worse, but we may not make things better. Often, the classes from the client
library of XYZ are on the Beam connector's API surface. In this case, we cannot
alter them, or you will get very confusing error messages "expected
SomeXYZClass but found SomeXYZClass" because they have nonequal class
signatures.
Would you be able to provide any more detail about which dependencies in
particular? Beam's current approach is to detect problematic dependencies and
attempt to create one global-to-Beam vendored version of the library. If we can
find a way to vendor a dependency in a connector, we could do that.
> Vendor or shade the dependencies of beam-sdks-java-io-google-cloud-platform
> ---------------------------------------------------------------------------
>
> Key: BEAM-8497
> URL: https://issues.apache.org/jira/browse/BEAM-8497
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Reporter: Julien Tournay
> Priority: Major
>
> Here at Spotify, we regularly face conflicting dependencies issues, usually
> caused by guava, grpc-* and netty-*. Looking at this bug tracker, it seems
> that we are not alone and that conflicting versions are a recurring issue.
> In {{beam-sdks-java-core,}} {{beam-runners-direct-java}} and
> {{beam-runners-google-cloud-dataflow-java}} those dependencies are vendored
> to avoid such issue.
> I suggest to do the same for {{beam-sdks-java-io-google-cloud-platform}}.
>
> FIY, here's the list of all the dependencies pulled by
> {{beam-sdks-java-io-google-cloud-platform}}:
> {{com.fasterxml.jackson.core:jackson-annotations:2.9.10:default}}
> {{com.fasterxml.jackson.core:jackson-core:2.9.10:default}}
> {{com.fasterxml.jackson.core:jackson-databind:2.9.10:default}}
> {{com.google.api:api-common:1.7.0:default}}
> {{com.google.api:gax:1.38.0:default}}
> {{com.google.api:gax-grpc:1.38.0:default}}
> {{com.google.api:gax-httpjson:0.52.0:default}}
> {{com.google.api-client:google-api-client:1.27.0:default}}
> {{com.google.api-client:google-api-client-jackson2:1.27.0:default}}
> {{com.google.api-client:google-api-client-java6:1.27.0:default}}
> {{com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta1:0.44.0:default}}
> {{com.google.api.grpc:grpc-google-cloud-bigtable-admin-v2:0.38.0:default}}
> {{com.google.api.grpc:grpc-google-cloud-bigtable-v2:0.38.0:default}}
> {{com.google.api.grpc:grpc-google-cloud-pubsub-v1:1.43.0:default}}
> {{com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:1.6.0:default}}
> {{com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:1.6.0:default}}
> {{com.google.api.grpc:grpc-google-cloud-spanner-v1:1.6.0:default}}
> {{com.google.api.grpc:grpc-google-common-protos:1.12.0:default}}
> {{com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.44.0:default}}
> {{com.google.api.grpc:proto-google-cloud-bigtable-admin-v2:0.38.0:default}}
> {{com.google.api.grpc:proto-google-cloud-bigtable-v2:0.44.0:default}}
> {{com.google.api.grpc:proto-google-cloud-datastore-v1:0.44.0:default}}
> {{com.google.api.grpc:proto-google-cloud-pubsub-v1:1.43.0:default}}
> {{com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:1.6.0:default}}
> {{com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:1.6.0:default}}
> {{com.google.api.grpc:proto-google-cloud-spanner-v1:1.6.0:default}}
> {{com.google.api.grpc:proto-google-common-protos:1.12.0:default}}
> {{com.google.api.grpc:proto-google-iam-v1:0.12.0:default}}
> {{com.google.apis:google-api-services-bigquery:v2-rev20181104-1.27.0:default}}
> {{com.google.apis:google-api-services-cloudresourcemanager:v1-rev20181015-1.27.0:default}}
> {{com.google.apis:google-api-services-pubsub:v1-rev20181105-1.27.0:default}}
> {{com.google.apis:google-api-services-storage:v1-rev20181109-1.27.0:default}}
> {{com.google.auth:google-auth-library-credentials:0.12.0:default}}
> {{com.google.auth:google-auth-library-oauth2-http:0.12.0:default}}
> {{com.google.auto.value:auto-value-annotations:1.6.3:default}}
> {{com.google.cloud:google-cloud-bigquerystorage:0.79.0-alpha:default}}
> {{com.google.cloud:google-cloud-bigtable:0.73.0-alpha:default}}
> {{com.google.cloud:google-cloud-bigtable-admin:0.73.0-alpha:default}}
> {{com.google.cloud:google-cloud-core:1.61.0:default}}
> {{com.google.cloud:google-cloud-core-grpc:1.61.0:default}}
> {{com.google.cloud:google-cloud-core-http:1.55.0:default}}
> {{com.google.cloud:google-cloud-spanner:1.6.0:default}}
> {{com.google.cloud.bigdataoss:gcsio:1.9.16:default}}
> {{com.google.cloud.bigdataoss:util:1.9.16:default}}
> {{com.google.cloud.bigtable:bigtable-client-core:1.8.0:default}}
> {{com.google.cloud.datastore:datastore-v1-proto-client:1.6.0:default}}
> {{com.google.code.findbugs:jsr305:3.0.2:default}}
> {{com.google.code.gson:gson:2.7:default}}
> {{com.google.errorprone:error_prone_annotations:2.3.2:default}}
> {{com.google.flogger:flogger:0.3.1:default}}
> {{com.google.flogger:flogger-system-backend:0.3.1:default}}
> {{com.google.flogger:google-extensions:0.3.1:default}}
> {{com.google.guava:failureaccess:1.0.1:default}}
> {{com.google.guava:guava:27.0.1-jre:default}}
> {{com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava:default}}
> {{com.google.http-client:google-http-client:1.27.0:default}}
> {{com.google.http-client:google-http-client-appengine:1.27.0:default}}
> {{com.google.http-client:google-http-client-jackson:1.20.0:default}}
> {{com.google.http-client:google-http-client-jackson2:1.27.0:default}}
> {{com.google.http-client:google-http-client-protobuf:1.20.0:default}}
> {{com.google.j2objc:j2objc-annotations:1.1:default}}
> {{com.google.oauth-client:google-oauth-client:1.27.0:default}}
> {{com.google.oauth-client:google-oauth-client-java6:1.27.0:default}}
> {{com.google.protobuf:protobuf-java:3.6.1:default}}
> {{com.google.protobuf:protobuf-java-util:3.6.1:default}}
> {{com.google.protobuf.nano:protobuf-javanano:3.0.0-alpha-5:default}}
> {{com.squareup.okhttp:okhttp:2.5.0:default}}
> {{com.squareup.okio:okio:1.13.0:default}}
> {{com.thoughtworks.paranamer:paranamer:2.7:default}}
> {{commons-codec:commons-codec:1.10:default}}
> {{commons-logging:commons-logging:1.2:default}}
> {{io.dropwizard.metrics:metrics-core:3.1.2:default}}
> {{io.grpc:grpc-all:1.17.1:default}}
> {{io.grpc:grpc-alts:1.17.1:default}}
> {{io.grpc:grpc-auth:1.17.1:default}}
> {{io.grpc:grpc-context:1.17.1:default}}
> {{io.grpc:grpc-core:1.17.1:default}}
> {{io.grpc:grpc-grpclb:1.17.1:default}}
> {{io.grpc:grpc-netty:1.17.1:default}}
> {{io.grpc:grpc-netty-shaded:1.17.1:default}}
> {{io.grpc:grpc-okhttp:1.17.1:default}}
> {{io.grpc:grpc-protobuf:1.17.1:default}}
> {{io.grpc:grpc-protobuf-lite:1.17.1:default}}
> {{io.grpc:grpc-protobuf-nano:1.17.1:default}}
> {{io.grpc:grpc-stub:1.17.1:default}}
> {{io.grpc:grpc-testing:1.17.1:default}}
> {{io.netty:netty-buffer:4.1.30.Final:default}}
> {{io.netty:netty-codec:4.1.30.Final:default}}
> {{io.netty:netty-codec-http:4.1.30.Final:default}}
> {{io.netty:netty-codec-http2:4.1.30.Final:default}}
> {{io.netty:netty-codec-socks:4.1.30.Final:default}}
> {{io.netty:netty-common:4.1.30.Final:default}}
> {{io.netty:netty-handler:4.1.30.Final:default}}
> {{io.netty:netty-handler-proxy:4.1.30.Final:default}}
> {{io.netty:netty-resolver:4.1.30.Final:default}}
> {{io.netty:netty-tcnative-boringssl-static:2.0.17.Final:default}}
> {{io.netty:netty-transport:4.1.30.Final:default}}
> {{io.opencensus:opencensus-api:0.17.0:default}}
> {{io.opencensus:opencensus-contrib-grpc-metrics:0.17.0:default}}
> {{io.opencensus:opencensus-contrib-grpc-util:0.17.0:default}}
> {{io.opencensus:opencensus-contrib-http-util:0.15.0:default}}
> {{javax.annotation:javax.annotation-api:1.2:default}}
> {{joda-time:joda-time:2.10.3:default}}
> {{junit:junit:4.12:default}}
> {{org.apache.avro:avro:1.8.2:default}}
> {{org.apache.beam:beam-model-job-management:2.16.0:default}}
> {{org.apache.beam:beam-model-pipeline:2.16.0:default}}
> {{org.apache.beam:beam-sdks-java-core:2.16.0:default}}
> {{org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:2.16.0:default}}
> {{org.apache.beam:beam-sdks-java-extensions-protobuf:2.16.0:default}}
> {{org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.16.0:default}}
> {{org.apache.beam:beam-vendor-bytebuddy-1_9_3:0.1:default}}
> {{org.apache.beam:beam-vendor-grpc-1_21_0:0.1:default}}
> {{org.apache.beam:beam-vendor-guava-26_0-jre:0.1:default}}
> {{org.apache.commons:commons-compress:1.8.1:default}}
> {{org.apache.commons:commons-lang3:3.5:default}}
> {{org.apache.httpcomponents:httpclient:4.5.5:default}}
> {{org.apache.httpcomponents:httpcore:4.4.9:default}}
> {{org.checkerframework:checker-qual:2.5.2:default}}
> {{org.codehaus.jackson:jackson-core-asl:1.9.13:default}}
> {{org.codehaus.jackson:jackson-mapper-asl:1.9.13:default}}
> {{org.codehaus.mojo:animal-sniffer-annotations:1.17:default}}
> {{org.hamcrest:hamcrest-core:1.3:default}}
> {{org.mockito:mockito-core:1.9.5:default}}
> {{org.objenesis:objenesis:1.0:default}}
> {{org.slf4j:slf4j-api:1.7.25:default}}
> {{org.threeten:threetenbp:1.3.3:default}}
> {{org.tukaani:xz:1.8:default}}
> {{org.xerial.snappy:snappy-java:1.1.4:default}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
