2018-04-26 7:54 GMT+02:00 Reuven Lax <re...@google.com>:
> This seems more than just Gradle, as they are using internal details in
> the parent pom (all the Google Cloud dependency version). Saying that
> nothing should break means that we can never rename or refactor anything in
> our poms because some user might be depending on it; that would be true on
> Maven just as much as on Gradle.
>
> Maybe this particular use is important enough that it needs to be
> maintained, I'm not sure. However we also need to decide what our stable
> pom interface is for users.
>
Browsing pom hierarchy to grab artifacts (including properties or not) and
plugin configurations (it is common to extract compiler config and shade
configs) is widely spread cause maven is the standard exchange format so I
hope beam will respect that at least.
The other usage is to check the dependencies between modules and determine
the impact of a change ("what is the common part" ~ parent+bom, "what does
it impact if we change that dep"). It can be key for beam to keep that to
ensure upstream projects can determine some of their impacts statically
through mavenrepo tools.
>
> Reuven
>
> On Wed, Apr 25, 2018 at 9:34 PM Romain Manni-Bucau <rmannibu...@gmail.com>
> wrote:
>
>> Hmm, gradle move not supposed to break any user and there are other
>> parents usages, as mentionned previously, so not sure beam can really
>> bypass it. Maven discussed consumer vs build pom since years for that
>> reason ;).
>>
>> Le 26 avr. 2018 01:59, "Eric Beach" <ebe...@google.com> a écrit :
>>
>>> Thanks Scott and Ben.
>>>
>>> This makes sense, and of course the same error happens when I tried
>>> beam-parent instead of beam-examples-parent.
>>>
>>> There is a big benefit for supporting explicit dependence on parent
>>> (either beam-examples-parent or beam-parent). In the project I'm involved
>>> in, I have Dataflow API (to query metrics for completed jobs), Google Cloud
>>> Storage, and Cloud Spanner (to execute queries outside of Beam).
>>>
>>>
>>> On Wed, Apr 25, 2018 at 7:55 PM Scott Wegner <sweg...@google.com> wrote:
>>>
>>>> Interesting. So two things:
>>>>
>>>> 1) As you correctly deduced, we switched to producing snapshots from
>>>> the Gradle build, which doesn't include parent pom's. As a result, the
>>>> parent pom snapshot you're referencing is stale. We should probably remove
>>>> it from the repository to prevent accidental usage until we get this sorted
>>>> out.
>>>>
>>>> 2) Depending on the parent pom directly isn't an intentionally
>>>> supported scenario, but your use-case seems valid: successful execution on
>>>> Dataflow requires specific version sets for Google Cloud dependencies. And
>>>> those dependency versions are conveniently defined as properties in the
>>>> beam-parent
>>>> pom
>>>> <https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-parent/2.5.0-SNAPSHOT/beam-parent-2.5.0-20180408.070218-37.pom>
>>>> .
>>>>
>>>> Note that simply generating parent pom's from Gradle wouldn't fix this;
>>>> we need to explicitly generate these version properties in the pom file.
>>>>
>>>> We could publish a special GCP parent pom just for this purpose.
>>>> Although forcing a parent relationship seems unnecessary; is there a better
>>>> convention within Maven for importing a set property values?
>>>>
>>>> On Wed, Apr 25, 2018 at 4:37 PM Eric Beach <ebe...@google.com> wrote:
>>>>
>>>>> If I run $ mvn clean compile exec:java ... on my project, I get the
>>>>> following error stack trace:
>>>>>
>>>>> java.lang.reflect.InvocationTargetException
>>>>>
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(
>>>>> NativeMethodAccessorImpl.java:62)
>>>>>
>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>>>> DelegatingMethodAccessorImpl.java:43)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>
>>>>> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(
>>>>> ExecJavaMojo.java:294)
>>>>>
>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>
>>>>> Caused by: java.lang.NoClassDefFoundError: com/google/api/services/
>>>>> clouddebugger/v2/CloudDebugger
>>>>>
>>>>> at java.lang.Class.getDeclaredMethods0(Native Method)
>>>>>
>>>>> at java.lang.Class.privateGetDeclaredMethods(Class.java:2703)
>>>>>
>>>>> at java.lang.Class.getDeclaredMethod(Class.java:2130)
>>>>>
>>>>> at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(
>>>>> InstanceBuilder.java:206)
>>>>>
>>>>> at org.apache.beam.sdk.util.InstanceBuilder.build(
>>>>> InstanceBuilder.java:162)
>>>>>
>>>>> at org.apache.beam.sdk.PipelineRunner.fromOptions(
>>>>> PipelineRunner.java:55)
>>>>>
>>>>> at org.apache.beam.sdk.Pipeline.create(Pipeline.java:150)
>>>>>
>>>>> at com.google.cloud.pontem.CloudSpannerDatabaseBackup.main(
>>>>> CloudSpannerDatabaseBackup.java:359)
>>>>>
>>>>> ... 6 more
>>>>>
>>>>> Caused by: java.lang.ClassNotFoundException: com.google.api.services.
>>>>> clouddebugger.v2.CloudDebugger
>>>>>
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>>>>
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>
>>>>>
>>>>> When I dig deeper, the root problem seems to be that my project is
>>>>> pulling in v2-rev8-1.22.0 instead of v2-rev233-1.23.0 (changing commit
>>>>> here
>>>>> <https://github.com/apache/beam/commit/66fcf4bff19cda5831465ccedbbaa6fbaa648234#diff-600376dffeb79835ede4a0b285078036>
>>>>> ).
>>>>>
>>>>> In fact, all the version changes in this commit
>>>>> <https://github.com/apache/beam/commit/66fcf4bff19cda5831465ccedbbaa6fbaa648234#diff-600376dffeb79835ede4a0b285078036>
>>>>> are missing.
>>>>>
>>>>> My project's pom.xml contains the following:
>>>>>
>>>>> <parent>
>>>>>
>>>>> <artifactId>beam-examples-parent</artifactId>
>>>>>
>>>>> <groupId>org.apache.beam</groupId>
>>>>>
>>>>> <version>2.5.0-SNAPSHOT</version>
>>>>>
>>>>> </parent>
>>>>>
>>>>> <modelVersion>4.0.0</modelVersion>
>>>>>
>>>>>
>>>>> <groupId>com.google.cloud</groupId>
>>>>>
>>>>> <artifactId>xyz</artifactId>
>>>>>
>>>>>
>>>>> <repositories>
>>>>>
>>>>> <repository>
>>>>>
>>>>> <id>apache.snapshots</id>
>>>>>
>>>>> <name>Apache Development Snapshot Repository</name>
>>>>>
>>>>> <url>https://repository.apache.org/content/
>>>>> repositories/snapshots/</url>
>>>>>
>>>>> <releases>
>>>>>
>>>>> <enabled>false</enabled>
>>>>>
>>>>> </releases>
>>>>>
>>>>> <snapshots>
>>>>>
>>>>> <enabled>true</enabled>
>>>>>
>>>>> </snapshots>
>>>>>
>>>>> </repository>
>>>>>
>>>>> </repositories>
>>>>>
>>>>> My first round of attempts to debug included removing ~/.m2/ and a
>>>>> number of other techniques.
>>>>>
>>>>> When I look at https://repository.apache.org/content/repositories/
>>>>> snapshots, I notice that beam-parent with 2.5.0-SNAPSHOT
>>>>> <https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-parent/2.5.0-SNAPSHOT/>
>>>>> is
>>>>> very outdated and so all the version numbers inherited from the POM are
>>>>> wrong (see https://repository.apache.org/content/
>>>>> repositories/snapshots/org/apache/beam/beam-parent/2.5.0-SNAPSHOT/).
>>>>> If you look at https://repository.apache.org/content/repositories/
>>>>> snapshots/org/apache/beam/ you will see that the beam-parent is
>>>>> significantly old.
>>>>>
>>>>> So, it seems imperative that the snapshot be rebuilt before the
>>>>> finalized 2.5.0 version is set or any projects depending on the Beam
>>>>> parent
>>>>> will break. (I depend upon the Beam parent because trying to keep the
>>>>> versions of all the different Google projects in sync is a total
>>>>> nightmare).
>>>>>
>>>>>
>>>>> On Wed, Apr 25, 2018 at 6:58 PM Chamikara Jayalath <
>>>>> chamik...@google.com> wrote:
>>>>>
>>>>>> Ccing Eric for providing more context.
>>>>>>
>>>>>> Thanks,
>>>>>> Cham
>>>>>>
>>>>>> On Wed, Apr 25, 2018 at 3:38 PM Scott Wegner <sweg...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Do you have any more context on how they were using the parent pom?
>>>>>>> In the Gradle build, instead of using a parent pom hierarchy we are
>>>>>>> embedding the complete set of dependencies and metadata in each
>>>>>>> generated
>>>>>>> pom. They should be functionally equivalent.
>>>>>>>
>>>>>>> On Wed, Apr 25, 2018 at 3:09 PM Chamikara Jayalath <
>>>>>>> chamik...@google.com> wrote:
>>>>>>>
>>>>>>>> At least one user was depending on 'beam-parent' and ran into
>>>>>>>> issues due to dependency update https://github.com/apache/
>>>>>>>> beam/pull/5046 not being reflected there. I'm not sure if this is
>>>>>>>> a usage pattern that we should discourage.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Cham
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Apr 25, 2018 at 2:36 PM Alan Myrvold <amyrv...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> The gradle build is currently only generating pom files when there
>>>>>>>>> is a jar file.
>>>>>>>>> What are these parent poms used for? Do all the parent poms need
>>>>>>>>> to be generated or just the top level beam-parent?
>>>>>>>>>
>>>>>>>>> On Wed, Apr 25, 2018 at 2:07 PM Chamikara Jayalath <
>>>>>>>>> chamik...@google.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks. Should this be a 2.5.0 blocker ?
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 25, 2018 at 2:05 PM Alan Myrvold <amyrv...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I think this corresponds with the move from maven to gradle
>>>>>>>>>>> snapshot releases.
>>>>>>>>>>> Issue logged as https://issues.apache.org/jira/browse/BEAM-4170
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Apr 25, 2018 at 1:57 PM Chamikara Jayalath <
>>>>>>>>>>> chamik...@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Seems like we are not building some of the SNAPSHOT artifacts
>>>>>>>>>>>> since 4/9/2018.
>>>>>>>>>>>> For example:
>>>>>>>>>>>>
>>>>>>>>>>>> https://repository.apache.org/content/repositories/
>>>>>>>>>>>> snapshots/org/apache/beam/beam-parent/2.5.0-SNAPSHOT/
>>>>>>>>>>>> https://repository.apache.org/content/repositories/
>>>>>>>>>>>> snapshots/org/apache/beam/beam-examples-parent/2.5.0-SNAPSHOT/
>>>>>>>>>>>>
>>>>>>>>>>>> Users who depend on these artifacts are getting outdated
>>>>>>>>>>>> dependencies.
>>>>>>>>>>>>
>>>>>>>>>>>> Any idea why ?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Cham
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>
>>>>>>>
>>>>>>> Got feedback? http://go/swegner-feedback
>>>>>>>
>>>>>> --
>>>>
>>>>
>>>> Got feedback? http://go/swegner-feedback
>>>>
>>>
