[
https://issues.apache.org/jira/browse/BEAM-4087?focusedWorklogId=236320&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-236320
]
ASF GitHub Bot logged work on BEAM-4087:
----------------------------------------
Author: ASF GitHub Bot
Created on: 02/May/19 14:37
Start Date: 02/May/19 14:37
Worklog Time Spent: 10m
Work Description: adude3141 commented on issue #8255: [BEAM-4087]
implement configurable dependency versions
URL: https://github.com/apache/beam/pull/8255#issuecomment-488698460
Thanks @lukecwik for looking into this and your feedback. Of course this are
very valid concerns.
> Won't this change allow for subprojects to increase the diamond dependency
problem for users when these are released as Maven artifacts?
Well, probably yes. At least if meant in the sense of making it easier for
the beam developer to mess it up.
What this change does, is first expose the configured versions to be
readable for beam (sub)projects. This is something which seems to be required.
At least some workarounds are already implemented to get this information. We
might argue, that in some of the examples given it would be better to include
that other libs also directly into the list of shared libraries. Not sure,
whether that would solve all issues which require reading of those 'shared
version'. Do you see any reason to not expose this information at least
read-only?
Now to the other changes.
#### Overwrites of common shared versions.
I assume, that's what worries you most. And, indeed, if subporjects start
randomly changing versions, just because some dev feels like 'having a newer
version is better' we are in deep trouble. So we should be very clear on that
you *should not* overwrite a common shared version. Well, unless you have a
very good reason. Unsure, whether [1] qualifies, but let's construct some
hypothetical example.
Assume having n IOs, with A_IO only working with somelib < X.Y, whereas B_IO
does work only with version >= X.Y. All others IOs are compatible with any
version of somelib. It is obvious, that A_IO and B_IO will never work together.
We could now pin a version globally, and either drop support for A_IO or B_IO,
pin no version at all or allow to overwrite the version for e.g. A_IO. I
believe third option to be the most flexible here from user perspective.
Or consider an IO growing old, getting less support on its dependencies. Do
we really want to be forced to either drop support of that IO or keep version
of some shared transitive dependency fixed to that old version, only because
any newer version stopped working with that IO?
During implementation I was also concerned about that 'easy overwrite' and
considered implementing something like
```
beam.versions {
guava "24.1-jre" overwrite
}
```
throwing an exception if flag `overwrite` is missing but version already
exists. So that developer is forced to think twice, whether she really want to
do that.
#### Adding to that shared version list.
This, of course will only add on project, so could be considered useless. On
the other hand, we might establish some checks later on which help detecting
version issues. Also it is likely, that we continue later by exposing also
java.library and corresponding version enforcements.
#### Passing version by CLI
This is mainly sugar. Usually we do not want to use that (although something
like this is already implemented, see example above)
It might be helpful to easily check, whether a different dependency version
is working with beam and might be upgraded (without editing BeamModulePlugin).
Also, which I would consider helpful, is enabling a user to rebuild and test
an artefact with different dependencies. Most likely the user has external
constraints which makes it impossible to run with dependencies chosen by beam
(e.g. different spark cluster version, other IOs backend version etc) and
having that possibility to check might be helpful.
> One of the benefits of using a common shared list of dependencies is that
we effectively test these subprojects with a known good set of library
versions. This increases the interoperability of Beam subprojects when released
as Maven artifacts. Getting more and more stuff into the shared list is a good
thing for our users.
Yes. This PR does not intend to change this. Unfortunately, the user is
still forced to solve diamond dependency problems on her project as beams
enforcement is of course not active on users project. But at least she can
trust beam to be tested and interoperable if she would be able to enforce to
beams transitive dependency versions (which might or might not be possible).
> For BEAM-4087, you could create multiple kafka configurations (like
testRuntimeKafka0_11_0) and run tests over each configuration. This is
effectively what we do with runners and examples/java here
Is this really the case? Wouldn't that mean, that I either have to create a
separate project which does not use BeamModulePlugin, as plugin enforces
version on all configurations [2], or remove the dependency you want to chan ge
from that common shared list? And a 'solution' like [3] to get that running
seems also awkward (and obviously a maintainance burden, netty version got
updated already on beams shared list). Note, that this is 'working' only as
netty-all is not part of common libraries.
Honestly, I am unsure, whether the proposed solution on BEAM-7003 [4] is any
better.
[1]
https://lists.apache.org/thread.html/4de4f53a62e862f249ccb27001fa41a38db478cf1c3fe9d15522c2f5@%3Cdev.beam.apache.org%3E
[2]
https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1166-L1181
[3]
https://github.com/apache/beam/blob/b419ac468724fe2bef0fdaa9b5509b154270f453/examples/java/build.gradle#L82-L84
[4]
https://issues.apache.org/jira/browse/BEAM-7003?focusedCommentId=16815436&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815436
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 236320)
Time Spent: 2h 20m (was: 2h 10m)
> Gradle build does not allow to overwrite versions of provided dependencies
> --------------------------------------------------------------------------
>
> Key: BEAM-4087
> URL: https://issues.apache.org/jira/browse/BEAM-4087
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
> Affects Versions: 2.5.0
> Reporter: Ismaël Mejía
> Assignee: Michael Luckey
> Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> In order to test modules with provided dependencies in maven we can execute
> for example for Kafka `mvn verify -Prelease -Dkafka.clients.version=0.9.0.1
> -pl 'sdks/java/io/kafka'` However we don't have an equivalent way to do this
> with gradle because the version of the dependencies are defined locally and
> not in the gradle.properties.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
