@Valentyn: concretely any user can PR and be part of that process so anyone can do it wrong (me first) @Luskasz, Hennking: fine but what do we do? last months prooved beam cant maintain 2 systems, easier with that state is then to drop gradle since it is a 0 investment compared to the opposite
Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <https://www.packtpub.com/application-development/java-ee-8-high-performance> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik <lc...@google.com>: > Romain, from the previous discussions several people agreed that running a > fixit that migrated Maven to Gradle over a 1 or 2 day period was worthwhile > but there was nobody in the community with the time commitment to organize > and run it so the status quo plan remained where the Gradle migration will > happen incrementally. > > > On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde <hero...@google.com> wrote: > >> My understanding was the same as Ismaël's. I don't think breaking the >> build with a large known gaps (but not fully known cost) is practical. >> Also, most items in the jira are not even assigned yet. >> >> >> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <rmannibu...@gmail.com> >> wrote: >> >>> Not really Ismaël, this thread was about to do it at once and have 1 day >>> to fix it all. >>> >>> As mentionned at the very beginning nobody maintains the 2 system so it >>> must stop after months so either we drop maven or gradle *at once* >>> or we keep a state where each dev does what he wants and the build >>> system just doesn't work. >>> >>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía <ieme...@gmail.com>: >>> >>>> I don't think that removing all maven descriptors was the expected >>>> path, no ? Or even a good idea at this moment. >>>> >>>> I understood that what we were going to do was to replace >>>> incrementally the CI until we cover the whole maven functionality and >>>> then remove it, from looking at the JIRA ticket >>>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from >>>> covering the complete maven functionality in particular for the >>>> release part that could be the biggest pain point. >>>> >>>> >>>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau >>>> <rmannibu...@gmail.com> wrote: >>>> > hey guys, >>>> > >>>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on >>>> monday? >>>> > >>>> > >>>> > Romain Manni-Bucau >>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>> > >>>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles <k...@google.com>: >>>> >> >>>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <lc...@google.com> >>>> wrote: >>>> >>> >>>> >>> Based upon your description it seems as though you would rather >>>> have a >>>> >>> way to run existing postcommits without it impacting >>>> dashboard/health >>>> >>> stats/notifications/.... (We have just run the PostCommits on PRs >>>> for >>>> >>> additional validation (like upgrading the Dataflow container >>>> image)). >>>> >> >>>> >> >>>> >> Yes, that is exactly what I have described. >>>> >> >>>> >>> I don't think that keeping the current Java PreCommit as a proxy >>>> for the >>>> >>> the Java PostCommit is the right way to go but I also don't have >>>> the time to >>>> >>> implement what your actually asking for. >>>> >> >>>> >> >>>> >> Mostly I thought this might be very easy based on the fact that they >>>> are >>>> >> nearly identical. If not, oh well. >>>> >> >>>> >> Kenn >>>> >> >>>> >> >>>> >>> It seems more likely that migrating the PostCommit to Gradle will >>>> be less >>>> >>> work then adding the functionality but your argument where the >>>> PreCommit is >>>> >>> a proxy for the Java PostCommit also applies to the ValidatesRunner >>>> >>> PostCommits and so forth requiring even more migration to happen >>>> before you >>>> >>> don't have to worry about maintaining Maven/breaking post commits. >>>> >>> >>>> >>> I'm fine with leaving both the Java/Gradle PreCommits running for >>>> now and >>>> >>> hopefully as more of the PostCommits are migrated off we will be >>>> able to >>>> >>> remove it. >>>> >>> >>>> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles <k...@google.com> >>>> wrote: >>>> >>>> >>>> >>>> Separate history (for easy dashboarding, health stats, etc) and >>>> >>>> notification (email to dev@ for postcommits, nothing for >>>> precommits) for pre >>>> >>>> & post commit targets. >>>> >>>> >>>> >>>> A post commit failure is always a problem to be triaged at high >>>> >>>> priority, while a precommit failure is just a natural occurrence. >>>> >>>> >>>> >>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <lc...@google.com> >>>> wrote: >>>> >>>>> >>>> >>>>> Ken, I'm probably not seeing something but how does using the >>>> PreCommit >>>> >>>>> as a proxy improve upon just running the post commit via the >>>> phrase it >>>> >>>>> already supports ('Run Java PostCommit')? >>>> >>>>> >>>> >>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <k...@google.com> >>>> >>>>> wrote: >>>> >>>>>> >>>> >>>>>> Indeed, we've already had the discussion a couple of times and I >>>> think >>>> >>>>>> the criteria are clearly met. Incremental progress is a good >>>> thing and we >>>> >>>>>> shouldn't block it. >>>> >>>>>> >>>> >>>>>> OTOH I see where Romain is coming from and I have a good example >>>> that >>>> >>>>>> supports a slightly different action. Consider >>>> >>>>>> https://github.com/apache/beam/pull/4740 which fixes some >>>> errors in how we >>>> >>>>>> use dependency mechanisms. >>>> >>>>>> >>>> >>>>>> This PR is green except that I need to fix some Maven pom >>>> slightly >>>> >>>>>> more. That is throwaway work. I would love to just not have to >>>> do it. But >>>> >>>>>> removing the precommit does not actually make the PR OK to >>>> merge. It would >>>> >>>>>> cause postcommits to fail. >>>> >>>>>> >>>> >>>>>> We can hope such situations are rare. I think I tend to be hit >>>> by this >>>> >>>>>> more often than most, as I work with the project build health >>>> quite a bit. >>>> >>>>>> >>>> >>>>>> Here is a proposal to support these things: instead of deleting >>>> the >>>> >>>>>> job in #4814, move it to not run automatically but only via a >>>> phrase. In >>>> >>>>>> fact, you could migrate it to be the manually-invoked version of >>>> the >>>> >>>>>> postcommit job as we've discussed a couple times. Then if >>>> someone is working >>>> >>>>>> on the build in something like #4740 they can invoke it manually. >>>> >>>>>> >>>> >>>>>> Kenn >>>> >>>>>> >>>> >>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <lc...@google.com> >>>> wrote: >>>> >>>>>>> >>>> >>>>>>> Based upon the criteria that was discussed on the mailing >>>> list[1], I >>>> >>>>>>> would agree with Kenn about merging PR/4814 (drop Java Maven >>>> precommit). >>>> >>>>>>> >>>> >>>>>>> 1: >>>> >>>>>>> https://lists.apache.org/thread.html/ >>>> 7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@% >>>> 3Cdev.beam.apache.org%3E >>>> >>>>>>> >>>> >>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau >>>> >>>>>>> <rmannibu...@gmail.com> wrote: >>>> >>>>>>>> >>>> >>>>>>>> Hi Kenneth, >>>> >>>>>>>> >>>> >>>>>>>> For now maven covers the full needs of beam. If we start to >>>> have >>>> >>>>>>>> this kind of PR we become dependent of the 2 builds which is >>>> what this >>>> >>>>>>>> thread is about avoiding so tempted to say it must be a PR >>>> drop completely >>>> >>>>>>>> maven or nothing as mentionned before. >>>> >>>>>>>> >>>> >>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <k...@google.com> a >>>> écrit : >>>> >>>>>>>>> >>>> >>>>>>>>> I would like to briefly re-focus this discussion and suggest >>>> that >>>> >>>>>>>>> we merge https://github.com/apache/beam/pull/4814. >>>> >>>>>>>>> >>>> >>>>>>>>> The only material objection I've heard is that it means the >>>> >>>>>>>>> precommit no longer tests exactly what is built for release. >>>> It is a valid >>>> >>>>>>>>> concern, but we have mvn postcommit so the coverage is not >>>> lost. The >>>> >>>>>>>>> benefits of quicker reviews and less Jenkins congestion seem >>>> worth it to me. >>>> >>>>>>>>> >>>> >>>>>>>>> Kenn >>>> >>>>>>>>> >>>> >>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau >>>> >>>>>>>>> <rmannibu...@gmail.com> wrote: >>>> >>>>>>>>>> >>>> >>>>>>>>>> @Luskasz: not sure Im the best to host it since I know more >>>> gradle >>>> >>>>>>>>>> internals that user interface/ecosystem but happy to help. >>>> Will also require >>>> >>>>>>>>>> a "sudo" merger for this day to merge fixes asap - guess we >>>> can bypass >>>> >>>>>>>>>> reviews or have a fast cycle plan for this day to avoid it >>>> to be a week? >>>> >>>>>>>>>> >>>> >>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <k...@google.com> a >>>> écrit : >>>> >>>>>>>>>> >>>> >>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better than >>>> >>>>>>>>>> Maven, and what I mean is that I have added enough hints >>>> that it works OOTB >>>> >>>>>>>>>> already. The rest of my instructions are just how you should >>>> override >>>> >>>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly just >>>> about storing >>>> >>>>>>>>>> files outside the git clone. >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> Hmm it is always super slow here and not as integrated as >>>> maven >>>> >>>>>>>>>> which is smooth thanks to the combination of idea build and >>>> maven plugin >>>> >>>>>>>>>> config read. Import is also faster cause it just reads meta >>>> and doesnt run >>>> >>>>>>>>>> anything. Hope it is a "a few times" issues at the moment >>>> but not yet sure. >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> @Łukasz - yea I just mean the ones that I find in my email >>>> and >>>> >>>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/ >>>> which are the ones >>>> >>>>>>>>>> you classify as "totally not working". I would love to >>>> disable these. >>>> >>>>>>>>>> >>>> >>>>>>>>>> On the subject of running things on a pending PR - we should >>>> not >>>> >>>>>>>>>> run postcommit jobs on PRs. We should make separate >>>> (optional) precommit >>>> >>>>>>>>>> jobs that run the same build commands. That will give a more >>>> clear history >>>> >>>>>>>>>> and allow trivial configuration of which jobs deserve alert >>>> emails and which >>>> >>>>>>>>>> are not a problem. This is easy but I've been waiting to do >>>> it after Gradle >>>> >>>>>>>>>> migration. >>>> >>>>>>>>>> >>>> >>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik < >>>> lc...@google.com> >>>> >>>>>>>>>> wrote: >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> Romain, would you like to host/plan/run the Gradle fixit >>>> day? >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath >>>> >>>>>>>>>>> <chamik...@google.com> wrote: >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle. >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> Agree with what Łukasz said. Some of these performance >>>> tests are >>>> >>>>>>>>>>>> new and are flaky due to other issues that were discovered >>>> during the >>>> >>>>>>>>>>>> process of adding the test. >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> I think the high level blocker is updating performance >>>> testing >>>> >>>>>>>>>>>> framework to use Gradle. This will involve adding >>>> Gradle-based logic for >>>> >>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and updating >>>> PerfKitBenchmarker >>>> >>>>>>>>>>>> to issue a Gradle command for running the benchmark [2]. >>>> First task will be >>>> >>>>>>>>>>>> to find the work needed here and updating the relevant >>>> JIRA [3]. >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> Thanks, >>>> >>>>>>>>>>>> Cham >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> [1] >>>> >>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/ >>>> google-cloud-platform/pom.xml#L79 >>>> >>>>>>>>>>>> [2] >>>> >>>>>>>>>>>> https://github.com/GoogleCloudPlatform/ >>>> PerfKitBenchmarker/blob/master/perfkitbenchmarker/ >>>> beam_benchmark_helper.py >>>> >>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251 >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy >>>> >>>>>>>>>>>> <lukasz.gaj...@gmail.com> wrote: >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles < >>>> k...@google.com>: >>>> >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ >>>> and our >>>> >>>>>>>>>>>>>> failure spam level the performance tests are mostly not >>>> healthy anyhow. So >>>> >>>>>>>>>>>>>> is there any high level blocker to switching them or is >>>> it just someone >>>> >>>>>>>>>>>>>> sitting down with each one? >>>> >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> I thought I could share my point of view here as I am >>>> working >>>> >>>>>>>>>>>>> on the Performance Test part for a while now. I wouldn't >>>> say those are >>>> >>>>>>>>>>>>> "mostly not healthy". The situation is as follows: >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> - totally not working: Python, Spark performance tests >>>> (don't >>>> >>>>>>>>>>>>> know why, I'm not maintaining those tests. Should we >>>> disable them?) >>>> >>>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance Test. >>>> It's >>>> >>>>>>>>>>>>> seems to be flaky mostly due to: >>>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798 >>>> >>>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO, Compressed >>>> >>>>>>>>>>>>> Text, TFRecordIO >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> Also, some test failures are caused due to pending PRs (we >>>> >>>>>>>>>>>>> sometimes run concrete tests to check if PRs won't break >>>> them). This also >>>> >>>>>>>>>>>>> causes failures sometimes. >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> I can help with switching the Performance tests to Gradle >>>> as >>>> >>>>>>>>>>>>> this part seems to be free to take. >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> Łukasz >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>> >>>> >>>>> >>>> >>> >>>> > >>>> >>> >>>
