On Fri, Feb 2, 2018 at 7:18 AM, Romain Manni-Bucau <rmannibu...@gmail.com>
wrote:

> Don't forget beam doesn't support much behind it (mainly only a few
> ParameterizedType due the usage code path) so it is mainly only about
> handling parameterized types and typevariables recursively. Not a lot of
> work. Main concern being it is in the API so using a shade as an API is
> never a good idea, in particular cause the shade can be broken and requires
> to setup clirr or things like that and when it breaks you are done and need
> to fork it anyway. Limiting the dependencies for an API - as beam is - is
> always saner even if it requires a small fork of code.
>

The main thing that TypeToken is used for is capturing generics that are
lost by Java reflection. It is a bit tricky, actually.

Kenn



>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
> 2018-02-02 15:49 GMT+01:00 Kenneth Knowles <k...@google.com>:
>
>> On Fri, Feb 2, 2018 at 6:41 AM, Romain Manni-Bucau <rmannibu...@gmail.com
>> > wrote:
>>
>>>
>>>
>>> 2018-02-02 15:37 GMT+01:00 Kenneth Knowles <k...@google.com>:
>>>
>>>> Another couple:
>>>>
>>>>  - User-facing TypeDescriptor is a very thin wrapper on Guava's
>>>> TypeToken
>>>>
>>>
>>> Technically reflect Type is enough
>>>
>>
>> If you want to try to remove TypeToken from underneath TypeDescriptor, I
>> have no objections as long as you expand the test suite to cover all the
>> functionality where we trust TypeToken's tests. Good luck :-)
>>
>> Kenn
>>
>>
>>>
>>>
>>>>  - ImmutableList and friends and their builders are very widely used
>>>> and IMO still add a lot for readability and preventing someone coming along
>>>> and adding mistakes to a codebase
>>>>
>>>
>>> Sugar but not required. When you compare the cost of a shade versus of
>>> duplicating the parts we need there is no real match IMHO.
>>>
>>>
>>>>
>>>> So considering it all, I would keep a vendored Guava (but also move off
>>>> where we can, and also have our own improvements). I hope it will be a
>>>> near-empty build file to generate it so not a maintenance burden.
>>>>
>>>> Kenn
>>>>
>>>> On Thu, Feb 1, 2018 at 8:44 PM, Kenneth Knowles <k...@google.com> wrote:
>>>>
>>>>> Nice. This sounds like a great idea (or two?) and goes along with what
>>>>> I just started for futures.
>>>>>
>>>>> Guava: filed https://issues.apache.org/jira/browse/BEAM-3606 and
>>>>> assigned to Ismaël :-) and converted my futures thing to a subtask.
>>>>>
>>>>> Specific things for our micro guava:
>>>>>
>>>>>  - checkArgumentNotNull can throw the right exception
>>>>>  - our own Optional because Java's is not Serializable
>>>>>  - futures combinators since many are missing, especially Java's don't
>>>>> do exceptions right
>>>>>
>>>>> Protobuf: didn't file an issue because I'm not sure
>>>>>
>>>>> I was wondering if pre-shading works. We really need to get rid of it
>>>>> from public APIs in a 100% reliable way. It is also a problem for 
>>>>> Dataflow.
>>>>> I was wondering if one approach is to pre-shade gcpio-protobuf-java,
>>>>> gcpio-protobuf-java-util, gcpio-grpc-java, etc. Anything that needs to 
>>>>> take
>>>>> a Message object. (and do the same for beam-model-protobuf-java since the
>>>>> model bits have to depend on each other but nothing else).
>>>>>
>>>>> Kenn
>>>>>
>>>>> On Thu, Feb 1, 2018 at 1:56 AM, Ismaël Mejía <ieme...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Huge +1 to get rid of Guava!
>>>>>>
>>>>>> This solves annoying dependency issues for some IOs and allow us to
>>>>>> get rid of the shading that makes current jars bigger than they should
>>>>>> be.
>>>>>>
>>>>>> We can create our own 'micro guava' package with some classes for
>>>>>> things that are hard to migrate, or that we  prefer to still have like
>>>>>> the check* methods for example. Given the size of the task we should
>>>>>> probably divide it into subtasks, more important is to get rid of it
>>>>>> for 'sdks/java/core'. We can then attack other areas afterwards.
>>>>>>
>>>>>> Other important idea would be to get rid of Protobuf in public APIs
>>>>>> like GCPIO and to better shade it from leaking into the runners. An
>>>>>> unexpected side effect of this is a leak of netty via gRPC/protobuf
>>>>>> that is byting us for the Spark runner, but well that's worth a
>>>>>> different discussion.
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 1, 2018 at 10:08 AM, Romain Manni-Bucau
>>>>>> <rmannibu...@gmail.com> wrote:
>>>>>> > a map of list is fine and not a challenge we'll face long I hope ;)
>>>>>> >
>>>>>> >
>>>>>> > Romain Manni-Bucau
>>>>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>>>>>> >
>>>>>> > 2018-02-01 9:40 GMT+01:00 Reuven Lax <re...@google.com>:
>>>>>> >>
>>>>>> >> Not sure we'll be able to replace them all. Things like guava
>>>>>> Table and
>>>>>> >> Multimap don't have great replacements in Java8.
>>>>>> >>
>>>>>> >> On Wed, Jan 31, 2018 at 10:11 PM, Jean-Baptiste Onofré <
>>>>>> j...@nanthrax.net>
>>>>>> >> wrote:
>>>>>> >>>
>>>>>> >>> +1, it was on my TODO for a while waiting the Java8 update.
>>>>>> >>>
>>>>>> >>> Regards
>>>>>> >>> JB
>>>>>> >>>
>>>>>> >>> On 02/01/2018 06:56 AM, Romain Manni-Bucau wrote:
>>>>>> >>> > Why not dropping guava for all beam codebase? With java 8 it is
>>>>>> quite
>>>>>> >>> > easy to do
>>>>>> >>> > it and avoid a bunch of conflicts. Did it in 2 projects with
>>>>>> quite a
>>>>>> >>> > good result.
>>>>>> >>> >
>>>>>> >>> > Le 1 févr. 2018 06:50, "Lukasz Cwik" <lc...@google.com
>>>>>> >>> > <mailto:lc...@google.com>> a écrit :
>>>>>> >>> >
>>>>>> >>> >     Make sure to include the guava version in the artifact name
>>>>>> so that
>>>>>> >>> > we can
>>>>>> >>> >     have multiple vendored versions.
>>>>>> >>> >
>>>>>> >>> >     On Wed, Jan 31, 2018 at 9:16 PM, Kenneth Knowles <
>>>>>> k...@google.com
>>>>>> >>> >     <mailto:k...@google.com>> wrote:
>>>>>> >>> >
>>>>>> >>> >         I didn't have time for this, but it just bit me. We
>>>>>> definitely
>>>>>> >>> > have
>>>>>> >>> >         Guava on the API surface of runner support code in ways
>>>>>> that
>>>>>> >>> > get
>>>>>> >>> >         incompatibly shaded. I will probably start "1a" by
>>>>>> making a
>>>>>> >>> > shaded
>>>>>> >>> >         library org.apache.beam:vendored-guava and starting to
>>>>>> use it.
>>>>>> >>> > It sounds
>>>>>> >>> >         like there is generally unanimous support for that much,
>>>>>> >>> > anyhow.
>>>>>> >>> >
>>>>>> >>> >         Kenn
>>>>>> >>> >
>>>>>> >>> >         On Wed, Dec 13, 2017 at 8:31 AM, Aljoscha Krettek
>>>>>> >>> > <aljos...@apache.org
>>>>>> >>> >         <mailto:aljos...@apache.org>> wrote:
>>>>>> >>> >
>>>>>> >>> >             Thanks Ismaël for bringing up this discussion again!
>>>>>> >>> >
>>>>>> >>> >             I would be in favour of 1) and more specifically of
>>>>>> 1a)
>>>>>> >>> >
>>>>>> >>> >             Aljoscha
>>>>>> >>> >
>>>>>> >>> >
>>>>>> >>> >>             On 12. Dec 2017, at 18:56, Lukasz Cwik <
>>>>>> lc...@google.com
>>>>>> >>> >>             <mailto:lc...@google.com>> wrote:
>>>>>> >>> >>
>>>>>> >>> >>             You can always run tests on post shaded artifacts
>>>>>> instead
>>>>>> >>> >> of the
>>>>>> >>> >>             compiled classes, it just requires us to change
>>>>>> our maven
>>>>>> >>> >> surefire
>>>>>> >>> >>             / gradle test configurations but it is true that
>>>>>> most IDEs
>>>>>> >>> >> would
>>>>>> >>> >>             behave better with a dependency jar unless you
>>>>>> delegate
>>>>>> >>> >> all the
>>>>>> >>> >>             build/test actions to the build system and then it
>>>>>> won't
>>>>>> >>> >> matter.
>>>>>> >>> >>
>>>>>> >>> >>             On Mon, Dec 11, 2017 at 9:05 PM, Kenneth Knowles
>>>>>> >>> >> <k...@google.com
>>>>>> >>> >>             <mailto:k...@google.com>> wrote:
>>>>>> >>> >>
>>>>>> >>> >>                 There's also, with additional overhead,
>>>>>> >>> >>
>>>>>> >>> >>                 1a) A relocated and shipped package for each
>>>>>> thing we
>>>>>> >>> >> want to
>>>>>> >>> >>                 relocate. I think this has also been tried
>>>>>> outside
>>>>>> >>> >> Beam...
>>>>>> >>> >>
>>>>>> >>> >>                 Pros:
>>>>>> >>> >>                 * all the pros of 1) plus no bloat beyond what
>>>>>> is
>>>>>> >>> >> necessary
>>>>>> >>> >>                 Cons:
>>>>>> >>> >>                 * abandons whitelist approach for public deps,
>>>>>> >>> >> reverting to
>>>>>> >>> >>                 blacklist approach for trouble things like
>>>>>> guava, so a
>>>>>> >>> >> bit
>>>>>> >>> >>                 less principled
>>>>>> >>> >>
>>>>>> >>> >>                 For both 1) and 1a) I would add:
>>>>>> >>> >>
>>>>>> >>> >>                 Pros:
>>>>>> >>> >>                 * clearly readable dependency since code will
>>>>>> `import
>>>>>> >>> >>                 org.apache.beam.private.guava21` and IDEs will
>>>>>> >>> >> understand it
>>>>>> >>> >>                 is a distinct lilbrary
>>>>>> >>> >>                 * can run tests on unpackaged classes, as long
>>>>>> as deps
>>>>>> >>> >> are
>>>>>> >>> >>                 shaded or provided as jars
>>>>>> >>> >>                 * no mysterious action at a distance from
>>>>>> inherited
>>>>>> >>> >> configuration
>>>>>> >>> >>                 Cons:
>>>>>> >>> >>                 * need to adjust imports
>>>>>> >>> >>
>>>>>> >>> >>                 Kenn
>>>>>> >>> >>
>>>>>> >>> >>                 On Mon, Dec 11, 2017 at 9:57 AM, Lukasz Cwik
>>>>>> >>> >> <lc...@google.com
>>>>>> >>> >>                 <mailto:lc...@google.com>> wrote:
>>>>>> >>> >>
>>>>>> >>> >>                     I would suggest that either we use:
>>>>>> >>> >>                     1) A common deps package containing shaded
>>>>>> >>> >> dependencies
>>>>>> >>> >>                     allows for
>>>>>> >>> >>                     Pros
>>>>>> >>> >>                     * doesn't require the user to build an
>>>>>> uber jar
>>>>>> >>> >>                     Risks
>>>>>> >>> >>                     * dependencies package will keep growing
>>>>>> even if
>>>>>> >>> >> something
>>>>>> >>> >>                     is or isn't needed by all of Apache Beam
>>>>>> leading
>>>>>> >>> >> to a
>>>>>> >>> >>                     large jar anyways negating any space
>>>>>> savings
>>>>>> >>> >>
>>>>>> >>> >>                     2) Shade within each module to a common
>>>>>> location
>>>>>> >>> >> like
>>>>>> >>> >>                     org.apache.beam.relocated.guava....
>>>>>> >>> >>                     Pros
>>>>>> >>> >>                     * you only get the shaded dependencies of
>>>>>> the
>>>>>> >>> >> things that
>>>>>> >>> >>                     are required
>>>>>> >>> >>                     * its one less dependency for users to
>>>>>> manage
>>>>>> >>> >>                     Risks
>>>>>> >>> >>                     * requires an uber jar to be built to get
>>>>>> the
>>>>>> >>> >> space
>>>>>> >>> >>                     savings (either by a user or a
>>>>>> distribution of
>>>>>> >>> >> Apache
>>>>>> >>> >>                     Beam) otherwise we negate any space
>>>>>> savings.
>>>>>> >>> >>
>>>>>> >>> >>                     If we either use a common relocation
>>>>>> scheme or a
>>>>>> >>> >>                     dependencies jar then each relocation
>>>>>> should
>>>>>> >>> >> specifically
>>>>>> >>> >>                     contain the version number of the package
>>>>>> because
>>>>>> >>> >> we would
>>>>>> >>> >>                     like to allow for us to be using two
>>>>>> different
>>>>>> >>> >> versions of
>>>>>> >>> >>                     the same library.
>>>>>> >>> >>
>>>>>> >>> >>                     For the common deps package approach,
>>>>>> should we
>>>>>> >>> >> check in
>>>>>> >>> >>                     code where the imports contain the
>>>>>> relocated
>>>>>> >>> >> location
>>>>>> >>> >>                     (e.g. import
>>>>>> org.apache.beam.guava.20.0.com
>>>>>> >>> >>
>>>>>> >>> >> <http://org.apache.beam.guava.20.0.com/>.google.common.colle
>>>>>> ct.ImmutableList)?
>>>>>> >>> >>
>>>>>> >>> >>
>>>>>> >>> >>                     On Mon, Dec 11, 2017 at 8:47 AM,
>>>>>> Jean-Baptiste
>>>>>> >>> >> Onofré
>>>>>> >>> >>                     <j...@nanthrax.net <mailto:j...@nanthrax.net>>
>>>>>> wrote:
>>>>>> >>> >>
>>>>>> >>> >>                         Thanks for bringing that back.
>>>>>> >>> >>
>>>>>> >>> >>                         Indeed guava is shaded in different
>>>>>> uber-jar.
>>>>>> >>> >> Maybe we
>>>>>> >>> >>                         can have a common deps module that we
>>>>>> include
>>>>>> >>> >> once
>>>>>> >>> >>                         (but the user will have to explicitly
>>>>>> define
>>>>>> >>> >> the dep) ?
>>>>>> >>> >>
>>>>>> >>> >>                         Basically, what do you propose for
>>>>>> protobuf
>>>>>> >>> >>                         (unfortunately, I don't see an
>>>>>> obvious) ?
>>>>>> >>> >>
>>>>>> >>> >>                         Regards
>>>>>> >>> >>                         JB
>>>>>> >>> >>
>>>>>> >>> >>
>>>>>> >>> >>                         On 12/11/2017 05:35 PM, Ismaël Mejía
>>>>>> wrote:
>>>>>> >>> >>
>>>>>> >>> >>                             Hello, I wanted to bring back this
>>>>>> subject
>>>>>> >>> >> because
>>>>>> >>> >>                             I think we should
>>>>>> >>> >>                             take action on this and at least
>>>>>> first
>>>>>> >>> >> have a
>>>>>> >>> >>                             shaded version of guava.
>>>>>> >>> >>                             I was playing with a toy project
>>>>>> and I did
>>>>>> >>> >> the
>>>>>> >>> >>                             procedure we use to
>>>>>> >>> >>                             submit jars to a Hadoop cluster via
>>>>>> >>> >> Flink/Spark
>>>>>> >>> >>                             which involves
>>>>>> >>> >>                             creating an uber jar and I
>>>>>> realized that
>>>>>> >>> >> the size
>>>>>> >>> >>                             of the jar was way
>>>>>> >>> >>                             bigger than I expected, and the
>>>>>> fact that
>>>>>> >>> >> we shade
>>>>>> >>> >>                             guava in every
>>>>>> >>> >>                             module contributes to this. I
>>>>>> found guava
>>>>>> >>> >> shaded on:
>>>>>> >>> >>
>>>>>> >>> >>                             sdks/java/core
>>>>>> >>> >>                             runners/core-construction-java
>>>>>> >>> >>                             runners/core-java
>>>>>> >>> >>                             model/job-management
>>>>>> >>> >>                             runners/spark
>>>>>> >>> >>                             sdks/java/io/hadoop-file-system
>>>>>> >>> >>                             sdks/java/io/kafka
>>>>>> >>> >>
>>>>>> >>> >>                             This means at least 6 times more
>>>>>> of the
>>>>>> >>> >> size it
>>>>>> >>> >>                             should which counts in
>>>>>> >>> >>                             around 15MB more (2.4MB*6 deps) of
>>>>>> extra
>>>>>> >>> >> weight
>>>>>> >>> >>                             that we can simply
>>>>>> >>> >>                             reduce by using a shaded version
>>>>>> of the
>>>>>> >>> >> library.
>>>>>> >>> >>
>>>>>> >>> >>                             I add this point to the previous
>>>>>> ones
>>>>>> >>> >> mentioned
>>>>>> >>> >>                             during the discussion
>>>>>> >>> >>                             because this goes against the end
>>>>>> user
>>>>>> >>> >> experience
>>>>>> >>> >>                             and affects us all
>>>>>> >>> >>                             (devs/users).
>>>>>> >>> >>
>>>>>> >>> >>                             Another question is if we should
>>>>>> shade
>>>>>> >>> >> (and how)
>>>>>> >>> >>                             protocol buffers
>>>>>> >>> >>                             because now with the portability
>>>>>> work we
>>>>>> >>> >> are
>>>>>> >>> >>                             exposing it closer to the
>>>>>> >>> >>                             end users. I say this because I
>>>>>> also found
>>>>>> >>> >> an
>>>>>> >>> >>                             issue while running a
>>>>>> >>> >>                             job on YARN with the spark runner
>>>>>> because
>>>>>> >>> >>                             hadoop-common includes
>>>>>> >>> >>                             protobuf-java 2 and I had to
>>>>>> explicitly
>>>>>> >>> >> provide
>>>>>> >>> >>                             protocol-buffers 3 as
>>>>>> >>> >>                             a dependency to be able to use
>>>>>> triggers
>>>>>> >>> >> (note the
>>>>>> >>> >>                             Spark runner
>>>>>> >>> >>                             translates them using some method
>>>>>> from
>>>>>> >>> >>                             runners/core-java). Since
>>>>>> >>> >>                             hadoop-common is provided in the
>>>>>> cluster
>>>>>> >>> >> with the
>>>>>> >>> >>                             older version of
>>>>>> >>> >>                             protobuf, I am afraid that this
>>>>>> will bite
>>>>>> >>> >> us in
>>>>>> >>> >>                             the future.
>>>>>> >>> >>
>>>>>> >>> >>                             Ismaël
>>>>>> >>> >>
>>>>>> >>> >>                             ps. There is already a JIRA for
>>>>>> that
>>>>>> >>> >> shading for
>>>>>> >>> >>                             protobuf on
>>>>>> >>> >>                             hadoop-common but this is not
>>>>>> coming until
>>>>>> >>> >> version
>>>>>> >>> >>                             3 is out.
>>>>>> >>> >>
>>>>>> >>> >> https://issues.apache.org/jira/browse/HADOOP-13136
>>>>>> >>> >>
>>>>>> >>> >> <https://issues.apache.org/jira/browse/HADOOP-13136>
>>>>>> >>> >>
>>>>>> >>> >>                             ps2. Extra curious situation is to
>>>>>> see
>>>>>> >>> >> that the
>>>>>> >>> >>                             dataflow-runner ends
>>>>>> >>> >>                             up having guava shaded twice via
>>>>>> its
>>>>>> >>> >> shaded version on
>>>>>> >>> >>                             core-construction-java.
>>>>>> >>> >>
>>>>>> >>> >>                             ps3. Of course this message means a
>>>>>> >>> >> de-facto +1 at
>>>>>> >>> >>                             least to do it for
>>>>>> >>> >>                             guava and evaluate it for other
>>>>>> libs.
>>>>>> >>> >>
>>>>>> >>> >>
>>>>>> >>> >>                             On Tue, Oct 17, 2017 at 7:29 PM,
>>>>>> Lukasz
>>>>>> >>> >> Cwik
>>>>>> >>> >>                             <lc...@google.com.invalid
>>>>>> >>> >>                             <mailto:lc...@google.com.invalid>>
>>>>>> wrote:
>>>>>> >>> >>
>>>>>> >>> >>                                 An issue to call out is how to
>>>>>> deal
>>>>>> >>> >> with our
>>>>>> >>> >>                                 generated code (.avro and
>>>>>> >>> >>                                 .proto) as I don't believe
>>>>>> those
>>>>>> >>> >> plugins allow
>>>>>> >>> >>                                 you to generate code using a
>>>>>> >>> >>                                 shaded package prefix on
>>>>>> imports.
>>>>>> >>> >>
>>>>>> >>> >>                                 On Tue, Oct 17, 2017 at 10:28
>>>>>> AM,
>>>>>> >>> >> Thomas Groh
>>>>>> >>> >>                                 <tg...@google.com.invalid
>>>>>> >>> >>                                 <mailto:tg...@google.com.inva
>>>>>> lid>>
>>>>>> >>> >>                                 wrote:
>>>>>> >>> >>
>>>>>> >>> >>                                     +1 to the goal. I'm hugely
>>>>>> in
>>>>>> >>> >> favor of not
>>>>>> >>> >>                                     doing the same shading work
>>>>>> >>> >>                                     every time for
>>>>>> dependencies we
>>>>>> >>> >> know we'll use.
>>>>>> >>> >>
>>>>>> >>> >>                                     This also means that if we
>>>>>> end up
>>>>>> >>> >> pulling
>>>>>> >>> >>                                     in transitive dependencies
>>>>>> we
>>>>>> >>> >>                                     don't want in any
>>>>>> particular
>>>>>> >>> >> module we can
>>>>>> >>> >>                                     avoid having to adjust our
>>>>>> >>> >>                                     repackaging strategy for
>>>>>> that
>>>>>> >>> >> module -
>>>>>> >>> >>                                     which I have run into
>>>>>> face-first
>>>>>> >>> >> in
>>>>>> >>> >>                                     the past.
>>>>>> >>> >>
>>>>>> >>> >>                                     On Tue, Oct 17, 2017 at
>>>>>> 9:48 AM,
>>>>>> >>> >> Kenneth
>>>>>> >>> >>                                     Knowles
>>>>>> <k...@google.com.invalid
>>>>>> >>> >>                                     <mailto:
>>>>>> k...@google.com.invalid>>
>>>>>> >>> >>                                     wrote:
>>>>>> >>> >>
>>>>>> >>> >>                                         Hi all,
>>>>>> >>> >>
>>>>>> >>> >>                                         Shading is a big part
>>>>>> of how
>>>>>> >>> >> we keep
>>>>>> >>> >>                                         our dependencies sane
>>>>>> in Beam.
>>>>>> >>> >> But
>>>>>> >>> >>                                         downsides: shading is
>>>>>> super
>>>>>> >>> >> slow,
>>>>>> >>> >>                                         causes massive jar
>>>>>> bloat, and
>>>>>> >>> >> kind of
>>>>>> >>> >>                                         hard to get right
>>>>>> because
>>>>>> >>> >> artifacts
>>>>>> >>> >>                                         and namespaces are not
>>>>>> 1-to-1.
>>>>>> >>> >>
>>>>>> >>> >>                                         I know that some
>>>>>> communities
>>>>>> >>> >>                                         distribute their own
>>>>>> shaded
>>>>>> >>> >>                                         distributions of
>>>>>> >>> >>                                         dependencies. I had a
>>>>>> thought
>>>>>> >>> >> about
>>>>>> >>> >>                                         doing something
>>>>>> similar that I
>>>>>> >>> >> wanted
>>>>>> >>> >>                                         to throw out there for
>>>>>> people
>>>>>> >>> >> to poke
>>>>>> >>> >>                                         holes in.
>>>>>> >>> >>
>>>>>> >>> >>                                         To set the scene, here
>>>>>> is how
>>>>>> >>> >> I view
>>>>>> >>> >>                                         shading:
>>>>>> >>> >>
>>>>>> >>> >>                                           - A module has public
>>>>>> >>> >> dependencies
>>>>>> >>> >>                                         and private
>>>>>> dependencies.
>>>>>> >>> >>                                           - Public deps are
>>>>>> used for
>>>>>> >>> >> data
>>>>>> >>> >>                                         interchange; users
>>>>>> must share
>>>>>> >>> >> these
>>>>>> >>> >>
>>>>>> >>> >>                                     deps.
>>>>>> >>> >>
>>>>>> >>> >>                                           - Private deps are
>>>>>> just
>>>>>> >>> >>                                         functionality and can
>>>>>> be
>>>>>> >>> >> hidden (in
>>>>>> >>> >>                                         our case,
>>>>>> >>> >>                                         relocated + bundled)
>>>>>> >>> >>                                           - It isn't
>>>>>> necessarily that
>>>>>> >>> >> simple,
>>>>>> >>> >>                                         because public and
>>>>>> private
>>>>>> >>> >> deps
>>>>>> >>> >>
>>>>>> >>> >>                                     might
>>>>>> >>> >>
>>>>>> >>> >>                                         interact in
>>>>>> higher-order ways
>>>>>> >>> >>                                         ("public" is
>>>>>> contagious)
>>>>>> >>> >>
>>>>>> >>> >>                                         Shading is an
>>>>>> implementation
>>>>>> >>> >> detail of
>>>>>> >>> >>                                         expressing these
>>>>>> >>> >> characteristics.
>>>>>> >>> >>
>>>>>> >>> >>                                     We
>>>>>> >>> >>
>>>>>> >>> >>                                         use shading selectively
>>>>>> >>> >> because of its
>>>>>> >>> >>                                         downsides I mentioned
>>>>>> above.
>>>>>> >>> >>
>>>>>> >>> >>                                         But what about this
>>>>>> idea:
>>>>>> >>> >> Introduce
>>>>>> >>> >>                                         shaded deps as a single
>>>>>> >>> >> separate
>>>>>> >>> >>                                         artifact.
>>>>>> >>> >>
>>>>>> >>> >>                                           -
>>>>>> sdks/java/private-deps:
>>>>>> >>> >> bundled
>>>>>> >>> >>                                         uber jar with relocated
>>>>>> >>> >> versions of
>>>>>> >>> >>                                         everything we want to
>>>>>> shade
>>>>>> >>> >>
>>>>>> >>> >>                                           - sdks/java/core and
>>>>>> >>> >>                                         sdks/java/harness: no
>>>>>> >>> >> relocation or
>>>>>> >>> >>                                         bundling -
>>>>>> >>> >>                                         depends on
>>>>>> >>> >>
>>>>>>  `beam-sdks-java-private-deps`
>>>>>> >>> >> and
>>>>>> >>> >>                                         imports like
>>>>>> >>> >>
>>>>>> >>> >> `org.apache.beam.sdk.private.com.google.common`
>>>>>> >>> >>                                         directly (this is what
>>>>>> >>> >>                                         they
>>>>>> >>> >>                                         are rewritten to
>>>>>> >>> >>
>>>>>> >>> >>                                         Some benefits
>>>>>> >>> >>
>>>>>> >>> >>                                           - much faster builds
>>>>>> of
>>>>>> >>> >> other modules
>>>>>> >>> >>                                           - only one shaded
>>>>>> uber jar
>>>>>> >>> >>                                           - rare/no rebuilds
>>>>>> of the
>>>>>> >>> >> uber jar
>>>>>> >>> >>                                           - can use maven
>>>>>> enforcer to
>>>>>> >>> >> forbid
>>>>>> >>> >>                                         imports like
>>>>>> com.google.common
>>>>>> >>> >>                                           - configuration all
>>>>>> in one
>>>>>> >>> >> place
>>>>>> >>> >>                                           - no automated
>>>>>> rewriting of
>>>>>> >>> >> our real
>>>>>> >>> >>                                         code, which has led to
>>>>>> some
>>>>>> >>> >> major
>>>>>> >>> >>                                         confusion
>>>>>> >>> >>                                           - easy to implement
>>>>>> >>> >> incrementally
>>>>>> >>> >>
>>>>>> >>> >>                                         Downsides:
>>>>>> >>> >>
>>>>>> >>> >>                                           - plenty of effort
>>>>>> work to
>>>>>> >>> >> get there
>>>>>> >>> >>                                           - unclear how many
>>>>>> different
>>>>>> >>> >> such
>>>>>> >>> >>                                         deps modules we need;
>>>>>> sharing
>>>>>> >>> >> them
>>>>>> >>> >>
>>>>>> >>> >>                                     could
>>>>>> >>> >>
>>>>>> >>> >>                                         get weird
>>>>>> >>> >>                                           - if we hit a
>>>>>> roadblock, we
>>>>>> >>> >> will
>>>>>> >>> >>                                         have committed a lot
>>>>>> of time
>>>>>> >>> >>
>>>>>> >>> >>                                         Just something I was
>>>>>> musing as
>>>>>> >>> >> I spent
>>>>>> >>> >>                                         another evening
>>>>>> waiting for
>>>>>> >>> >> slow
>>>>>> >>> >>                                         builds to try to
>>>>>> confirm
>>>>>> >>> >> changes to
>>>>>> >>> >>                                         brittle poms.
>>>>>> >>> >>
>>>>>> >>> >>                                         Kenn
>>>>>> >>> >>
>>>>>> >>> >>
>>>>>> >>> >>
>>>>>> >>> >>                         --
>>>>>> >>> >>                         Jean-Baptiste Onofré
>>>>>> >>> >>                         jbono...@apache.org
>>>>>> >>> >> <mailto:jbono...@apache.org>
>>>>>> >>> >>                         http://blog.nanthrax.net
>>>>>> >>> >> <http://blog.nanthrax.net/>
>>>>>> >>> >>                         Talend - http://www.talend.com
>>>>>> >>> >> <http://www.talend.com/>
>>>>>> >>> >>
>>>>>> >>> >>
>>>>>> >>> >>
>>>>>> >>> >>
>>>>>> >>> >
>>>>>> >>> >
>>>>>> >>> >
>>>>>> >>>
>>>>>> >>> --
>>>>>> >>> Jean-Baptiste Onofré
>>>>>> >>> jbono...@apache.org
>>>>>> >>> http://blog.nanthrax.net
>>>>>> >>> Talend - http://www.talend.com
>>>>>> >>
>>>>>> >>
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Reply via email to