I opened up PR/4992[1] to disable all maven based Jenkins jobs and leave them around so that people working on the migration can update them to be Gradle based. This also ensures that there is plenty of Jenkins capacity for testing migration to Gradle.
It is important for Committers to be aware that this is being disabled and to be careful about merging non Gradle migration fixit tasks. 1: https://github.com/apache/beam/pull/4992 On Mon, Apr 2, 2018 at 10:03 AM Lukasz Cwik <[email protected]> wrote: > Aljoscha, to add things to be shaded they need to be part of the "compile" > configuration and you also need to tell the shadow plugin to add these and > how to relocate them like this: > > https://github.com/apache/beam/blob/cc2022157865d44e8dba070a766de78853c508bc/sdks/java/core/build.gradle#L77 > > Romain, the subtasks of BEAM-3249 seem to be centered around: > 1) Migrating Jenkins > 2) Making the artifacts/release (shading, jars, javadoc all fall under > this) > > I would defer things that are optional like "ensure concurrency is not > hardcoded" till after the migration. > > On Sat, Mar 31, 2018 at 2:28 AM Romain Manni-Bucau <[email protected]> > wrote: > >> Thanks for the doc Lukasz >> >> So we do 3 groups? >> >> 1. Jenkins >> 2. Release >> 3. Outputs/behavior (shades, jars, javadoc, ensure concurrency is not >> hardcoded, ...) >> >> ? >> >> Happy to be in 3 or 2 (by preference) if you agree. >> >> Le 31 mars 2018 10:54, "Aljoscha Krettek" <[email protected]> a écrit : >> >>> Thanks Luke, that was helpful! >>> >>> I've been playing around with gradle a bit and now have a question: By >>> default the shadow plugin will include dependencies that are "compile" or >>> "runtime" (compile dependencies are by default also runtime dependencies) >>> into the shaded jar. Is that correct? The reason our build doesn't do that >>> is that we have custom include/exclude rules that prevent that from >>> happening. I just noticed that when I change the Flink dependencies from >>> shadow to compile in the Flink Runner build file the shaded jar will not >>> contain the Flink classes. >>> >>> Btw, I'm not saying that I want to include the Flink classes, I was just >>> messing around with things to get a feeling for how things work. >>> >>> On 31. Mar 2018, at 00:15, Lukasz Cwik <[email protected]> wrote: >>> >>> I have started a doc[1] containing a Gradle primer to help people be >>> more productive during the fixit day. Feel free to add/update comments and >>> content. >>> >>> 1: >>> https://docs.google.com/document/d/1EiTwEMD8FNhU4Ok6jthASpmK3-1hiAYzVTrdl8qBLrs/edit?usp=sharing >>> >>> On Fri, Mar 30, 2018 at 8:33 AM Reuven Lax <[email protected]> wrote: >>> >>>> Here is the Kanban board tracking all the current tasks. It looks like >>>> 7 of them have already been closed over the past two days, so we're down to >>>> 19! >>>> >>>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242 >>>> >>>> On Thu, Mar 29, 2018 at 11:21 PM Romain Manni-Bucau < >>>> [email protected]> wrote: >>>> >>>>> Yep - sorry if it was unclear. I know linux distro often do it (never >>>>> understood why though). >>>>> >>>>> >>>>> Romain Manni-Bucau >>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>> <http://rmannibucau.wordpress.com/> | Github >>>>> <https://github.com/rmannibucau> | LinkedIn >>>>> <https://www.linkedin.com/in/rmannibucau> | Book >>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance> >>>>> >>>>> 2018-03-30 7:54 GMT+02:00 Reuven Lax <[email protected]>: >>>>> >>>>>> >>>>>> >>>>>> On Thu, Mar 29, 2018 at 10:28 PM Romain Manni-Bucau < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> It was more about the drop of poms (same case as you for dataflow). >>>>>>> >>>>>> >>>>>> Ah - you're worried that some external users are building directly >>>>>> from the poms rather than using the published artifact. >>>>>> >>>>>> I think this is a valid concern, and I agree we should announce on >>>>>> users@ before (probably some time before) deleting the poms . >>>>>> >>>>>> >>>>>>> On that there is a missing but highly important task: gradle to mvn >>>>>>> descriptors. All the one I saw were corrupted poms so we must take care >>>>>>> of >>>>>>> that as part of the release work (I can work on it on the 3rd if you >>>>>>> want). >>>>>>> >>>>>>> Le 29 mars 2018 23:36, "Reuven Lax" <[email protected]> a écrit : >>>>>>> >>>>>>>> I don't mind notifying users@, but this does seem more interesting >>>>>>>> for dev@. We will continue to publish Maven artifacts from our >>>>>>>> Gradle build, so users are still free to use either Maven or Gradle. >>>>>>>> >>>>>>>> That being said, if this is interesting to users@ we can notify >>>>>>>> them as well. >>>>>>>> >>>>>>>> Reuven >>>>>>>> >>>>>>>> On Thu, Mar 29, 2018 at 1:45 PM Romain Manni-Bucau < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Le 29 mars 2018 21:20, "Reuven Lax" <[email protected]> a écrit : >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Mar 29, 2018 at 12:17 PM Romain Manni-Bucau < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Le 29 mars 2018 20:35, "Reuven Lax" <[email protected]> a écrit : >>>>>>>>>> >>>>>>>>>> 1. As Luke already mentioned, we should first have a subgoal of >>>>>>>>>> the Gradle jenkins jobs being equivalent to the Maven jobs. Hopefully >>>>>>>>>> toward the end of the day, we'll make this change. >>>>>>>>>> >>>>>>>>>> 2. Let's see how much progress we make on the third. There is a >>>>>>>>>> side problem we have here at Google - we have an internal product >>>>>>>>>> called >>>>>>>>>> Dataflow built on Beam, and the Dataflow build still depends on >>>>>>>>>> those pom >>>>>>>>>> files. I would request leaving the pom files around just a little bit >>>>>>>>>> longer even if Beam no longer needs them, just so that we don't >>>>>>>>>> breakd >>>>>>>>>> Dataflow (and I think we would do this for any community members >>>>>>>>>> with a >>>>>>>>>> similar issue). We will prioritize moving Dataflow ASAP, it's just >>>>>>>>>> that the >>>>>>>>>> people who will do so will also be in the April 3 fixit so it can't >>>>>>>>>> happen >>>>>>>>>> until after. I think the delay should only be one or two weeks to >>>>>>>>>> delete >>>>>>>>>> the poms (assuming that Beam is ready at the end of the day). >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Can you try to put a date of that and we will communicate on it >>>>>>>>>> publicly if anyone else does (i dont think but i didnt expect you to >>>>>>>>>> do it >>>>>>>>>> too ;)). >>>>>>>>>> >>>>>>>>> >>>>>>>>> What do you mean by communicate publicly? The dev list is already >>>>>>>>> cced here :) >>>>>>>>> >>>>>>>>> >>>>>>>>> Misses a (not too far) date and users@ pby to be safe ;) >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Mar 29, 2018 at 5:20 AM Romain Manni-Bucau < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Reuven, a few questions: >>>>>>>>>>> >>>>>>>>>>> 1. any inputs on how we can work on the jenkins part? Do we test >>>>>>>>>>> it live wiht "fake" PRs? >>>>>>>>>>> 2. What's the rational to not start by deleting the poms? Sounds >>>>>>>>>>> like it will be a day working on gradle and on the 4th we'll be >>>>>>>>>>> back on >>>>>>>>>>> maven >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Romain Manni-Bucau >>>>>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>>>>>>> <http://rmannibucau.wordpress.com/> | Github >>>>>>>>>>> <https://github.com/rmannibucau> | LinkedIn >>>>>>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book >>>>>>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance> >>>>>>>>>>> >>>>>>>>>>> 2018-03-29 4:46 GMT+02:00 Reuven Lax <[email protected]>: >>>>>>>>>>> >>>>>>>>>>>> Hi all, >>>>>>>>>>>> >>>>>>>>>>>> Last week we discussed having a "fixit" day for Gradle, and I >>>>>>>>>>>> volunteered to organize it. A number of people volunteered to >>>>>>>>>>>> help, from >>>>>>>>>>>> multiple organization. I'd like to say that it's great to see such >>>>>>>>>>>> a >>>>>>>>>>>> diverse set of people volunteering to help here - this is a great >>>>>>>>>>>> way to >>>>>>>>>>>> build community! Everyone who explicitly volunteered is directly >>>>>>>>>>>> cced on >>>>>>>>>>>> this email, though we'd love for more of the community to help. >>>>>>>>>>>> >>>>>>>>>>>> The agreed upon date is April 3. The top-level JIRA tracking >>>>>>>>>>>> this work is >>>>>>>>>>>> ttps://issues.apache.org/jira/browse/BEAM-3249 >>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-3249>, and we >>>>>>>>>>>> currently have 26 subtasks linked to it. I've created a Kanban >>>>>>>>>>>> board to >>>>>>>>>>>> track these issues, which I'll share out soon. We will use Slack >>>>>>>>>>>> the day of >>>>>>>>>>>> the fixit for collaboration and for questions. >>>>>>>>>>>> >>>>>>>>>>>> Two major goals for this fixit should be to 1. Remove Maven >>>>>>>>>>>> runs from our Jenkins executors and 2. to migrate our release >>>>>>>>>>>> process fully >>>>>>>>>>>> over to Gradle. A lot of work has already been done on 1., and >>>>>>>>>>>> we've made >>>>>>>>>>>> some progress on 2.. Slightly longer-term the goal is to delete >>>>>>>>>>>> all of the >>>>>>>>>>>> pom files; I'm not sure we'll get as far as completely deleting >>>>>>>>>>>> Maven in >>>>>>>>>>>> one day, but we should get within striking distance! >>>>>>>>>>>> >>>>>>>>>>>> Thanks in advance to everyone who's helping out! >>>>>>>>>>>> >>>>>>>>>>>> Reuven >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>> >>>
