[
https://issues.apache.org/jira/browse/BEAM-4087?focusedWorklogId=236893&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-236893
]
ASF GitHub Bot logged work on BEAM-4087:
----------------------------------------
Author: ASF GitHub Bot
Created on: 03/May/19 15:11
Start Date: 03/May/19 15:11
Worklog Time Spent: 10m
Work Description: adude3141 commented on issue #8255: [BEAM-4087]
implement configurable dependency versions
URL: https://github.com/apache/beam/pull/8255#issuecomment-489125415
Thanks, @lukecwik for your explanations. Unfortunately, I have difficulties
to follow your argumentation.
> > Thanks @lukecwik for looking into this and your feedback. Of course this
are very valid concerns.
> > > Won't this change allow for subprojects to increase the diamond
dependency problem for users when these are released as Maven artifacts?
> >
> >
> > Well, probably yes. At least if meant in the sense of making it easier
for the beam developer to mess it up.
> > What this change does, is first expose the configured versions to be
readable for beam (sub)projects. This is something which seems to be required.
At least some workarounds are already implemented to get this information. We
might argue, that in some of the examples given it would be better to include
that other libs also directly into the list of shared libraries. Not sure,
whether that would solve all issues which require reading of those 'shared
version'. Do you see any reason to not expose this information at least
read-only?
>
> No objections to read-only. What use cases did you find that need to know
the version?
I tried to already reference those in the first description of this PR.
E.g. creating a dependency just to get to the version
https://github.com/apache/beam/pull/8255/files#diff-dff580209e15df561338df48bf2dd8eb
or respecifying netty_version here
https://github.com/apache/beam/pull/8255/files#diff-7f8dd8dd2d2484735c8af1b7a0a9b09c
>
> > Now to the other changes.
> > #### Overwrites of common shared versions.
> > I assume, that's what worries you most. And, indeed, if subporjects
start randomly changing versions, just because some dev feels like 'having a
newer version is better' we are in deep trouble. So we should be very clear on
that you _should not_ overwrite a common shared version. Well, unless you have
a very good reason. Unsure, whether [1] qualifies, but let's construct some
hypothetical example.
> > Assume having n IOs, with A_IO only working with somelib < X.Y, whereas
B_IO does work only with version >= X.Y. All others IOs are compatible with any
version of somelib. It is obvious, that A_IO and B_IO will never work together.
We could now pin a version globally, and either drop support for A_IO or B_IO,
pin no version at all or allow to overwrite the version for e.g. A_IO. I
believe third option to be the most flexible here from user perspective.
> > Or consider an IO growing old, getting less support on its dependencies.
Do we really want to be forced to either drop support of that IO or keep
version of some shared transitive dependency fixed to that old version, only
because any newer version stopped working with that IO?
>
> I would suspect that we could do any of the following:
>
> 1. Decide that the library will no longer be in the shared list and allow
subprojects to choose specific versions.
Isn't this *pin no version at all*? Drawback here would be, that we not only
need to respecify the version on all affected leave project, we also loose the
forcing of the version done by BeamModulePlugin, which, if not done, will
result in gradles default resolution strategy of 'use latest' which in turn
will let us end up with possibly lots of different dependency versions on our
modules. Thereby reintroducing that diamond dependency problem we initially
wanted to minimise by including this dependency into that list.
Isn't that worse than only changing the version on a specific module,
thereby restricting diamond issues only if that module is in use?
> 2. Drop the IO module and keep the library in the shared list.
Sure. We can always drop an IO implementation. But what are our users
supposed to do, which business rely on that certain module? Should they fork?
Or not upgrade anymore? Or hope they will be able to use that old version IO
module in conjunction with e.g. newer version of core?
> 3. Update the old IO module to support the new version of the dependency
as well.
Seem, I was not clear here. I did not try to imply, the beam IO module
getting outdated, but the corresponding backend which we do access by some
client lib which has the corresponding transitive dependency. So we would need
to take over that (external) client lib. Which might be doable, but I certainly
prefer not to be required to go down that route.
>
> > During implementation I was also concerned about that 'easy overwrite'
and considered implementing something like
> > ```
> > beam.versions {
> > guava "24.1-jre" overwrite
> > }
> > ```
> >
> >
> > throwing an exception if flag `overwrite` is missing but version already
exists. So that developer is forced to think twice, whether she really want to
do that.
> > #### Adding to that shared version list.
> > This, of course will only add on project, so could be considered
useless. On the other hand, we might establish some checks later on which help
detecting version issues. Also it is likely, that we continue later by exposing
also java.library and corresponding version enforcements.
> > #### Passing version by CLI
> > This is mainly sugar. Usually we do not want to use that (although
something like this is already implemented, see example above)
> > It might be helpful to easily check, whether a different dependency
version is working with beam and might be upgraded (without editing
BeamModulePlugin).
> > Also, which I would consider helpful, is enabling a user to rebuild and
test an artefact with different dependencies. Most likely the user has external
constraints which makes it impossible to run with dependencies chosen by beam
(e.g. different spark cluster version, other IOs backend version etc) and
having that possibility to check might be helpful.
>
> The person will have to look into BeamModulePlugin to find the magic
string for the version override.
Not necessarily. While true currently, after turning those magic strings
into 'public api' they should certainly be properly documented.
By the way, I actually believe they should already be documented in some way
or the other, as those fixed versions do constitute some important information
not only to the devs, but also to the user. We might consider implementing some
reporting task here or some such.
> Editing the version number in the file seems easier then passing it on the
command line once I have already had to look for it in the BeamModulePlugin.
Although that might look easier, it is most likely not something you want to
do.
First, editing BeamModulePlugin (or any build.gradle) might end up in an
accidental commit on some unrelated PR. So I usually discourage people doing so.
Second, editing a required Plugin implies a full rebuild of the repository.
As I most likely want to do a full build after changing a common library
version, i.e. run a
```
./gradlew build
```
this might have heavy impact. E.g. on my machine changing spark version from
2.4.0 to 2.4.2 by editing the plugin takes about 30min to execute.
```
./gradlew build -Pspark.version=2.4.2
```
takes only 90s.
Now as you most likely also revert that change and might build again, those
numbers get even doubled.
>
> > > One of the benefits of using a common shared list of dependencies is
that we effectively test these subprojects with a known good set of library
versions. This increases the interoperability of Beam subprojects when released
as Maven artifacts. Getting more and more stuff into the shared list is a good
thing for our users.
> >
> >
> > Yes. This PR does not intend to change this. Unfortunately, the user is
still forced to solve diamond dependency problems on her project as beams
enforcement is of course not active on users project. But at least she can
trust beam to be tested and interoperable if she would be able to enforce to
beams transitive dependency versions (which might or might not be possible).
>
> We do [release a
BOM](https://github.com/apache/beam/blob/master/sdks/java/bom/build.gradle) to
help users with some parts of the diamond dependency problem. We could add the
shared list to this BOM which would help them with diamond dependency issues
since they would be getting the versions of libraries that we are suggesting
they use.
Sure. This might help the user. Most likely she is not only using beam reps,
though, so some resolution outside of beams scope has to be done anyway. And
currently I do not understand all implications if we decide to implement this.
>
> > > For BEAM-4087, you could create multiple kafka configurations (like
testRuntimeKafka0_11_0) and run tests over each configuration. This is
effectively what we do with runners and examples/java here
> >
> >
> > Is this really the case? Wouldn't that mean, that I either have to
create a separate project which does not use BeamModulePlugin, as plugin
enforces version on all configurations [2], or remove the dependency you want
to chan ge from that common shared list? And a 'solution' like [3] to get that
running seems also awkward (and obviously a maintainance burden, netty version
got updated already on beams shared list). Note, that this is 'working' only as
netty-all is not part of common libraries.
>
> Two alternatives could be to:
>
> 1. Remove kafka from the shared list allowing each subproject to set
whichever version they want
> 2. Use
[dependencies.create('org.apache.kafka:kafka-client:$version')](https://docs.gradle.org/current/dsl/org.gradle.api.artifacts.dsl.DependencyHandler.html#org.gradle.api.artifacts.dsl.DependencyHandler:create(java.lang.Object))
and add that to the [test runtime classpath in a test {}
block](https://github.com/apache/beam/blob/ce77db10cdd5f021f383721a90f30205aff0fabe/examples/java/build.gradle#L111).
>
Isn't that equivalent to the proposal on BEAM-7003?
> > Honestly, I am unsure, whether the proposed solution on BEAM-7003 [4] is
any better.
> > [1]
https://lists.apache.org/thread.html/4de4f53a62e862f249ccb27001fa41a38db478cf1c3fe9d15522c2f5@%3Cdev.beam.apache.org%3E
> > [2]
https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1166-L1181
> > [3]
> >
https://github.com/apache/beam/blob/b419ac468724fe2bef0fdaa9b5509b154270f453/examples/java/build.gradle#L82-L84
> >
> > [4]
https://issues.apache.org/jira/browse/BEAM-7003?focusedCommentId=16815436&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815436
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 236893)
Time Spent: 3h 10m (was: 3h)
> Gradle build does not allow to overwrite versions of provided dependencies
> --------------------------------------------------------------------------
>
> Key: BEAM-4087
> URL: https://issues.apache.org/jira/browse/BEAM-4087
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
> Affects Versions: 2.5.0
> Reporter: Ismaël Mejía
> Assignee: Michael Luckey
> Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> In order to test modules with provided dependencies in maven we can execute
> for example for Kafka `mvn verify -Prelease -Dkafka.clients.version=0.9.0.1
> -pl 'sdks/java/io/kafka'` However we don't have an equivalent way to do this
> with gradle because the version of the dependencies are defined locally and
> not in the gradle.properties.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
