That's really cool ! I'm thinking about the Spark runner where grpc deps are not easy to deal with.
Thanks ! Regards JB On 10/07/2018 23:09, Lukasz Cwik wrote: > With the merge of PR #5594[1], we started shading all gRPC / Protobuf > dependencies within all the modules that depended on the model/* > dependencies by vendoring them. The vendored versions are built and > packaged into the model jars (they should be separated out once I figure > out how to generate proto code using a shaded import path). Note that > this cleaned up several issues where we were incorrectly built shaded > jars without repackaging in some locations or the shading process was > corrupting the contents of some of the jars. > > Note that the majority of the code base (especially related to > portability) should be using imports under the > org.apache.beam.vendor.protobuf.v3 or org.apache.beam.vendor.grpc.v1 > paths. I have yet to figure out a clean way to get Intellij to recognize > these vendored paths. My only solution so far has been to manually add > one of the built model jars to the compile classpath of the module being > worked on in Intellij as described here[2]. I would greatly appreciate > some ideas on how to improve this integration because from a few > attempts configuring the intellij gradle pluglin scope sections didn't > produce the result that I was expecting. > > I also added a simple test > task /validateShadedJarDoesntLeakNonOrgApacheBeamClasses/ that validates > the shaded jar doesn't contain classes without repackaging which we > should apply to any module that performs shading to ensure that classes > are relocated and we don't accidentally expose stuff. I filed > BEAM-4753[3] to this end. > > Note, these are the following modules that still depend on protobuf that > are shaded away and could move to use a vendored variant of protobuf: > * sdks/java/core > * sdks/java/extensions/sql > > These modules expose protobuf because it is part of the API surface: > * sdks/java/extensions/protobuf > * sdks/java/io/google-cloud-platform (I believe that gRPC could be > shaded here but preferrably the IO module would do it so we wouldn't > have this maintenance burden.) > > 1: https://github.com/apache/beam/pull/5594 > 2: > https://stackoverflow.com/questions/1051640/correct-way-to-add-external-jars-lib-jar-to-an-intellij-idea-project > 3: https://issues.apache.org/jira/browse/BEAM-4753 -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com
