You shouldn't do :module:cleanTest. If that is necessary that's a major bug in the build.
Kenn On Fri, Apr 13, 2018 at 11:46 PM Romain Manni-Bucau <[email protected]> wrote: > There is a fake module xxx_test which should have the right classpath but > since idea compilation is messed up you will still have to run > :module:cleanTest :module:test --tests org...MyTest.myMethod, even with > idea which leads to the same latency as the command line :( > > Le 13 avr. 2018 22:23, "Eugene Kirpichov" <[email protected]> a écrit : > >> While we're talking about running tests in IntelliJ with Gradle... >> Anybody got advice on how to run a single NeedsRunner test in >> sdks-java-core, say, ParDoTest? With Maven, I used to just run the test in >> IntelliJ and specify "runners-direct-java" as the classpath; with Gradle, >> the best I got is to manually run the direct runner's needsRunnerTests task >> with specifying --tests=..., but it takes a long time, and IntelliJ treats >> that as just a gradle task run, not as a test run. >> >> On Fri, Apr 13, 2018 at 11:14 AM Reuven Lax <[email protected]> wrote: >> >>> Is there a Jira for this 3 second delay? Also you're initial complaint >>> was not about the 3 second delay, so it wasn't clear that's what you were >>> complaining about. >>> >>> Reuven >>> >>> On Fri, Apr 13, 2018 at 4:42 AM Romain Manni-Bucau < >>> [email protected]> wrote: >>> >>>> When you launch a test with gradle runner it launches gradle which >>>> makes loose 3s on a very fast computer and more on a slower (6 on my >>>> personal one which is already fast but not as much as my work one). We are >>>> 5 to see that regression at least. So there is a reason to not use the >>>> gradle runner if possible cause when you work and need to debug you are >>>> just stucked (that is why i switched back to mvn after 15mn, i was loosing >>>> to much time). >>>> >>>> Switching back to native idea test run would fix it but tests just dont >>>> work this way for me whatever setup i do :( - missing resources IIRC in >>>> idea out dir. >>>> >>>> Le 13 avr. 2018 00:07, "Reuven Lax" <[email protected]> a écrit : >>>> >>>> I also don't quite understand what your question is, and it appears >>>> like Dan spent considerable time trying to reproduce your issue. For the >>>> record, I have had no issues running tests via Gradle in IntelliJ for the >>>> past few weeks. >>>> >>>> Reuven >>>> >>>> On Thu, Apr 12, 2018 at 9:47 PM Daniel Oliveira <[email protected]> >>>> wrote: >>>> >>>>> Sorry Romain, I'm not quite sure what you're asking. Can you clarify? >>>>> >>>>> On Thu, Apr 12, 2018 at 12:22 PM Romain Manni-Bucau < >>>>> [email protected]> wrote: >>>>> >>>>>> Well you are the only one to not have the drawbacks to use it so >>>>>> maybe dont do it? I know Luke is in holidays but anyone else with the >>>>>> knowledge of why we nees that noise compared to idea native tooling/flow? >>>>>> >>>>>> Le 12 avr. 2018 20:16, "Daniel Oliveira" <[email protected]> a >>>>>> écrit : >>>>>> >>>>>>> Ah, I did not. Thanks Romain. >>>>>>> >>>>>>> I tried it again, restarting in between, and still had no >>>>>>> differences. Since it seems like there's no reason not to use "Gradle >>>>>>> Test >>>>>>> Runner", I'll mention it in the contributor's guide. >>>>>>> >>>>>>> On Thu, Apr 12, 2018 at 10:31 AM Romain Manni-Bucau < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> @Daniel: did you restart in between? Otherwise it does nothing. One >>>>>>>> launches JunitCoreRunner from idea and the other a gradle command. >>>>>>>> >>>>>>>> Le 12 avr. 2018 19:24, "Daniel Oliveira" <[email protected]> >>>>>>>> a écrit : >>>>>>>> >>>>>>>>> I think it depends on what exactly switching to "Gradle Test >>>>>>>>> Runner" from "Platform Test Runner" does. I tried it out on my >>>>>>>>> machine and >>>>>>>>> they seem to act identically to each other. The IntelliJ >>>>>>>>> documentation says >>>>>>>>> it determines what API to use to run the tests >>>>>>>>> <https://www.jetbrains.com/help/idea/runner.html>, so maybe it's >>>>>>>>> usefulness depends on the user's machine, in which case a note about >>>>>>>>> that >>>>>>>>> would be useful. Something like: "If your IDE has trouble running >>>>>>>>> tests via >>>>>>>>> IDEA shortcuts, try the following steps: [...]" >>>>>>>>> >>>>>>>>> On Thu, Apr 12, 2018 at 3:29 AM Alexey Romanenko < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Daniel, actually I did run it with default IDEA JUnit test >>>>>>>>>> runner. Then, in “Settings > Build, Execution, Deployment > Build >>>>>>>>>> Tools > >>>>>>>>>> Gradle > Runner" I selected “Gradle Test Runner” in “Run tests using” >>>>>>>>>> selectbox and it works ok when I run my tests with IDEA shortcuts. >>>>>>>>>> So, >>>>>>>>>> probably, we should add this details on >>>>>>>>>> https://beam.apache.org/contribute/intellij/ too. >>>>>>>>>> What do you think? >>>>>>>>>> >>>>>>>>>> WBR, >>>>>>>>>> Alexey >>>>>>>>>> >>>>>>>>>> On 11 Apr 2018, at 21:17, Daniel Oliveira <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Alexey, are you referring to tests run with "./gradlew >>>>>>>>>> :beam-runners-direct-java:needsRunnerTests"? That command works fine >>>>>>>>>> for me >>>>>>>>>> in both versions of IDEA, but I believe the same tests fail if you >>>>>>>>>> run them >>>>>>>>>> directly through "./gradlew test". >>>>>>>>>> >>>>>>>>>> However, I am having issues with a bunch of validatesRunner >>>>>>>>>> tests, mostly be caused by >>>>>>>>>> :beam-runners-google-cloud-dataflow-java:validatesRunner. Not sure >>>>>>>>>> if it's >>>>>>>>>> because of a code change or a gradle config, I'll keep looking into >>>>>>>>>> it. >>>>>>>>>> >>>>>>>>>> On Wed, Apr 11, 2018 at 11:01 AM Romain Manni-Bucau < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> I got tests running rrconfiguring gradle (which was setup for >>>>>>>>>>> another project but seems beam didnt like it) but latency is still >>>>>>>>>>> "high" >>>>>>>>>>> using gradle runner for tests (like Etienne said ~3s on an i7 with >>>>>>>>>>> 16G vs a >>>>>>>>>>> few ms with default idea test runner, would be great to solve that). >>>>>>>>>>> >>>>>>>>>>> I also find the integration quite fishy cause configurations are >>>>>>>>>>> customs so idea is kind of forced to propose your modukes 3 times >>>>>>>>>>> at least >>>>>>>>>>> when you select the classpath (x_test being generally the working >>>>>>>>>>> one). >>>>>>>>>>> >>>>>>>>>>> Also the false positive you get if you forget a cleanX is a bit >>>>>>>>>>> annoying, maybe we should force a clean for test or at least when >>>>>>>>>>> there is >>>>>>>>>>> a --tests to avoid gradle to not run it cause there is no diff. >>>>>>>>>>> >>>>>>>>>>> So it works but dev productivity is reduced a lot and it became >>>>>>>>>>> slow enough to think if you should do a contribution or not - at >>>>>>>>>>> least for >>>>>>>>>>> me. >>>>>>>>>>> >>>>>>>>>>> Le 11 avr. 2018 19:37, "Alexey Romanenko" < >>>>>>>>>>> [email protected]> a écrit : >>>>>>>>>>> >>>>>>>>>>>> I’ve managed to import a project as it’s described in >>>>>>>>>>>> documentation (starting from empty project) using Idea 2018 and >>>>>>>>>>>> run unit >>>>>>>>>>>> tests successfully. >>>>>>>>>>>> For some reasons, tests, that use DirectRunner to run a >>>>>>>>>>>> pipeline, were failed. >>>>>>>>>>>> >>>>>>>>>>>> WBR, >>>>>>>>>>>> Alexey >>>>>>>>>>>> >>>>>>>>>>>> On 11 Apr 2018, at 19:01, Daniel Oliveira < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi everyone, I was the one who initially wrote the PR with >>>>>>>>>>>> Idea instructions >>>>>>>>>>>> <https://github.com/apache/beam-site/pull/414>. I was using >>>>>>>>>>>> 2017.3 as well while writing it so all the instructions were >>>>>>>>>>>> tested on that >>>>>>>>>>>> version. I'll try testing the instructions on 2018 to see if I can >>>>>>>>>>>> reproduce the issues people are having. >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Apr 11, 2018 at 9:51 AM Lukasz Cwik <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I use 2017.3 and it has been reliable for me. I haven't tried >>>>>>>>>>>>> 2018 yet. >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Any of you using the idea 2018? the import works for me but >>>>>>>>>>>>>> then it is >>>>>>>>>>>>>> not as smooth as it seems for you. I'm just trying to see if >>>>>>>>>>>>>> it is a >>>>>>>>>>>>>> procedure thing or a version issue. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Romain Manni-Bucau >>>>>>>>>>>>>> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2018-04-11 17:28 GMT+02:00 Kenneth Knowles <[email protected]>: >>>>>>>>>>>>>> > The only reason I did "empty project then add a module" >>>>>>>>>>>>>> procedure was to get >>>>>>>>>>>>>> > all the IntelliJ files outside the source tree. IIRC >>>>>>>>>>>>>> directly creating from >>>>>>>>>>>>>> > existing sources didn't give the necessary configuration >>>>>>>>>>>>>> options. If you >>>>>>>>>>>>>> > don't care about being able to `git clean` then you can do >>>>>>>>>>>>>> the shorter >>>>>>>>>>>>>> > version. Also the particular UI for creation might have >>>>>>>>>>>>>> improved since I >>>>>>>>>>>>>> > tried it. I'll do it again. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > On the pom, since it is only generated for machine reading, >>>>>>>>>>>>>> why care if the >>>>>>>>>>>>>> > parent is inlined? I actually prefer to avoid coupling with >>>>>>>>>>>>>> things that you >>>>>>>>>>>>>> > have to go and look up. Using inheritance is OK for human >>>>>>>>>>>>>> edited poms >>>>>>>>>>>>>> > (actually IMO it is still a mistake) but it doesn't change >>>>>>>>>>>>>> the semantics of >>>>>>>>>>>>>> > a shipped pom if they are all immutable, which they should >>>>>>>>>>>>>> be. It is better >>>>>>>>>>>>>> > to have all the info right there. Is there an actually >>>>>>>>>>>>>> effective difference? >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Kenn >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > On Wed, Apr 11, 2018 at 6:53 AM Etienne Chauchot < >>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> Hi all, >>>>>>>>>>>>>> >> I just tested gradle environment from a fresh source clone >>>>>>>>>>>>>> with this >>>>>>>>>>>>>> >> procedure with just a tiny change: I used "new project >>>>>>>>>>>>>> from existing >>>>>>>>>>>>>> >> sources" rather than create empty project and then add >>>>>>>>>>>>>> module. >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> It works fine and junit runs from intellij also work. >>>>>>>>>>>>>> with gradle we pay >>>>>>>>>>>>>> >> a 2s delay (on my machine) before running the actual test >>>>>>>>>>>>>> for each run. This >>>>>>>>>>>>>> >> delay seems quite constant no matter the module: I have >>>>>>>>>>>>>> run all the tests at >>>>>>>>>>>>>> >> once in runner-core and a single test in another module >>>>>>>>>>>>>> with a similar >>>>>>>>>>>>>> >> delay. >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> I also tested a debug session from intellij and it runs >>>>>>>>>>>>>> fine also. >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> I'll do some more testing and keep you posted. >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> I still have some concerns about the potential difficulty >>>>>>>>>>>>>> for new >>>>>>>>>>>>>> >> contributors to have to learn gradle in addition to Beam >>>>>>>>>>>>>> itself comparing to >>>>>>>>>>>>>> >> maven which is more mainstream for java developers. That >>>>>>>>>>>>>> could be >>>>>>>>>>>>>> >> discouraging for ex for part-time contributors >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> Etienne >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> Le mardi 10 avril 2018 à 14:38 +0000, Lukasz Cwik a écrit : >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> beam-site PR/414 updates the instructions for using >>>>>>>>>>>>>> Intellij and how to >>>>>>>>>>>>>> >> import a module: >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> 1. Create an empty IntelliJ project outside of the Beam >>>>>>>>>>>>>> source tree. >>>>>>>>>>>>>> >> 2. Under Project Structure > Project, select a Project SDK. >>>>>>>>>>>>>> >> 3. Under Project Structure > Modules, click the + sign to >>>>>>>>>>>>>> add a module and >>>>>>>>>>>>>> >> select "Import Module". >>>>>>>>>>>>>> >> 1. Select the directory containing the Beam source >>>>>>>>>>>>>> tree. >>>>>>>>>>>>>> >> 2. Tick the "Import module from external model" button >>>>>>>>>>>>>> and select >>>>>>>>>>>>>> >> Gradle >>>>>>>>>>>>>> >> from the list. >>>>>>>>>>>>>> >> 3. Tick the following boxes. >>>>>>>>>>>>>> >> * Use auto-import >>>>>>>>>>>>>> >> * Create separate module per source set >>>>>>>>>>>>>> >> * Store generated project files externally >>>>>>>>>>>>>> >> * Use default gradle wrapper >>>>>>>>>>>>>> >> 4. Delegate build actions to Gradle by going to Settings > >>>>>>>>>>>>>> Build, >>>>>>>>>>>>>> >> Execution, >>>>>>>>>>>>>> >> Deployment > Build Tools > Gradle and checking >>>>>>>>>>>>>> "Delegate IDE build/run >>>>>>>>>>>>>> >> actions to gradle". >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré < >>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>> >> wrote: >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> That's a very important issue for contribution. Up to now, >>>>>>>>>>>>>> I used Maven >>>>>>>>>>>>>> >> for setup IntelliJ (and it works just fine). If we remove >>>>>>>>>>>>>> the pom.xml, >>>>>>>>>>>>>> >> we have to support Eclipse and IntelliJ "smoothly". >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> Let me try in IntelliJ. >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> Regards >>>>>>>>>>>>>> >> JB >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> On 10/04/2018 15:21, Romain Manni-Bucau wrote: >>>>>>>>>>>>>> >> > You dont have issue due to the build setup with that >>>>>>>>>>>>>> option. I get: >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> > avr. 10, 2018 3:20:10 PM >>>>>>>>>>>>>> >> > org.apache.beam.runners.direct.DirectTransformExecutor >>>>>>>>>>>>>> run >>>>>>>>>>>>>> >> > GRAVE: Error occurred within >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a >>>>>>>>>>>>>> >> > com.google.common.util.concurrent.ExecutionError: >>>>>>>>>>>>>> >> > java.lang.NoClassDefFoundError: >>>>>>>>>>>>>> net/bytebuddy/NamingStrategy >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> > ? >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> > Romain Manni-Bucau >>>>>>>>>>>>>> >> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | >>>>>>>>>>>>>> Book >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik <[email protected] >>>>>>>>>>>>>> >: >>>>>>>>>>>>>> >> >> I have found that the simplest setup is to delegate the >>>>>>>>>>>>>> build/test >>>>>>>>>>>>>> >> >> actions >>>>>>>>>>>>>> >> >> to Gradle. This allows you to run unit tests very >>>>>>>>>>>>>> easily and since its >>>>>>>>>>>>>> >> >> in >>>>>>>>>>>>>> >> >> the same manner that Gradle would have, you know that >>>>>>>>>>>>>> if its passing it >>>>>>>>>>>>>> >> >> will >>>>>>>>>>>>>> >> >> pass on the command line and on Jenkins. Here is one >>>>>>>>>>>>>> site that >>>>>>>>>>>>>> >> >> discusses how >>>>>>>>>>>>>> >> >> to set this up: >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> >> >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau >>>>>>>>>>>>>> >> >> <[email protected]> >>>>>>>>>>>>>> >> >> wrote: >>>>>>>>>>>>>> >> >>> >>>>>>>>>>>>>> >> >>> What's the plan to make idea supporting gradle on beam >>>>>>>>>>>>>> project? Do we >>>>>>>>>>>>>> >> >>> import the workaround mentionned in >>>>>>>>>>>>>> >> >>> https://youtrack.jetbrains.com/issue/IDEA-175172? >>>>>>>>>>>>>> >> >>> For the ones who didn't see this issue in action: idea >>>>>>>>>>>>>> will compile in >>>>>>>>>>>>>> >> >>> out/ instead of build/ and you will just miss all the >>>>>>>>>>>>>> resources you >>>>>>>>>>>>>> >> >>> need like some SPI registration which are used by all >>>>>>>>>>>>>> our registrar => >>>>>>>>>>>>>> >> >>> no way to run tests in idea without hacking the >>>>>>>>>>>>>> configuration quite >>>>>>>>>>>>>> >> >>> deeply :( >>>>>>>>>>>>>> >> >>> >>>>>>>>>>>>>> >> >>> Romain Manni-Bucau >>>>>>>>>>>>>> >> >>> @rmannibucau | Blog | Old Blog | Github | LinkedIn | >>>>>>>>>>>>>> Book >>>>>>>>>>>>>> >> >>> >>>>>>>>>>>>>> >> >>> >>>>>>>>>>>>>> >> >>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot < >>>>>>>>>>>>>> [email protected]>: >>>>>>>>>>>>>> >> >>>> As a gradle beginner, I could not agree more ! >>>>>>>>>>>>>> >> >>>> +1 >>>>>>>>>>>>>> >> >>>> Etienne >>>>>>>>>>>>>> >> >>>> Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste >>>>>>>>>>>>>> Onofré a écrit : >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> Hi all, >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> I did multiple gradle build since last week and I >>>>>>>>>>>>>> would like to share >>>>>>>>>>>>>> >> >>>> one of my concern: it's about the communities. >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> If I think our users won't see any change for them >>>>>>>>>>>>>> due to Gradle >>>>>>>>>>>>>> >> >>>> build >>>>>>>>>>>>>> >> >>>> (I think that most of our users will still use Maven >>>>>>>>>>>>>> with artifacts >>>>>>>>>>>>>> >> >>>> provided by Gradle), I'm more concerned by the dev >>>>>>>>>>>>>> community and the >>>>>>>>>>>>>> >> >>>> contribution. >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> Maven is well known and straight forward for a large >>>>>>>>>>>>>> part of >>>>>>>>>>>>>> >> >>>> potential >>>>>>>>>>>>>> >> >>>> contributors. I think we have to keep in mind that we >>>>>>>>>>>>>> still have to >>>>>>>>>>>>>> >> >>>> grow >>>>>>>>>>>>>> >> >>>> up our contributors community. >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> Today, maybe I'm wrong, but I have the feeling that >>>>>>>>>>>>>> gradle build is >>>>>>>>>>>>>> >> >>>> not >>>>>>>>>>>>>> >> >>>> straight forward (build.gradle includes >>>>>>>>>>>>>> build_rules.gradle, gathering >>>>>>>>>>>>>> >> >>>> all taks all together). >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> I would like to add a task in the gradle "migration" >>>>>>>>>>>>>> process: >>>>>>>>>>>>>> >> >>>> simplify >>>>>>>>>>>>>> >> >>>> the gradle structure and files, and document this. >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> I know we already have a Jira about the documentation >>>>>>>>>>>>>> part, but I >>>>>>>>>>>>>> >> >>>> would >>>>>>>>>>>>>> >> >>>> like to "polish" and use a clean structure for the >>>>>>>>>>>>>> Gradle resources. >>>>>>>>>>>>>> >> >>>> As >>>>>>>>>>>>>> >> >>>> already quickly discussed, I think that having one >>>>>>>>>>>>>> gradle file per >>>>>>>>>>>>>> >> >>>> tasks >>>>>>>>>>>>>> >> >>>> in the .gradle directory would be helpful. >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> The goal is really to simplify the contribution. >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> Do you agree if I add a Jira about "Gradle polish" ? >>>>>>>>>>>>>> >> >>>> Thoughts ? >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> Regards >>>>>>>>>>>>>> >> >>>> JB >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> On 07/04/2018 04:52, Scott Wegner wrote: >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> Here's an end-of-day update on migration work: >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> * Snapshot unsigned dailies and signed release builds >>>>>>>>>>>>>> are working >>>>>>>>>>>>>> >> >>>> (!!). >>>>>>>>>>>>>> >> >>>> PR/5048 [1] merges changes from Luke's branch >>>>>>>>>>>>>> >> >>>> * python precommit failing... will investigate >>>>>>>>>>>>>> python precommit >>>>>>>>>>>>>> >> >>>> Monday >>>>>>>>>>>>>> >> >>>> * All Precommits are gradle only >>>>>>>>>>>>>> >> >>>> * All Postcommits except performance tests and >>>>>>>>>>>>>> Java_JDK_Versions_Test >>>>>>>>>>>>>> >> >>>> use gradle (after PR/5047 [2] merged) >>>>>>>>>>>>>> >> >>>> * Nightly snapshot release using gradle is ready; >>>>>>>>>>>>>> needs PR/5048 to be >>>>>>>>>>>>>> >> >>>> merged before switching >>>>>>>>>>>>>> >> >>>> * ValidatesRunner_Spark failing consistently; >>>>>>>>>>>>>> investigating >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> Thanks for another productive day of hacking. I'll >>>>>>>>>>>>>> pick up again on >>>>>>>>>>>>>> >> >>>> Monday. >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> [1] https://github.com/apache/beam/pull/5048 >>>>>>>>>>>>>> >> >>>> [2] https://github.com/apache/beam/pull/5047 >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau >>>>>>>>>>>>>> >> >>>> <[email protected] <mailto:[email protected]>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> Why building a zip per runner which its stack >>>>>>>>>>>>>> and just pointing >>>>>>>>>>>>>> >> >>>> out >>>>>>>>>>>>>> >> >>>> on that zip and let beam lazy load the runner: >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> --runner=LazyRunner --lazyRunnerDir=... >>>>>>>>>>>>>> --lazyRunnerOptions=... >>>>>>>>>>>>>> >> >>>> (or >>>>>>>>>>>>>> >> >>>> the fromSystemProperties() if it gets merged a >>>>>>>>>>>>>> day ;)) >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> Le 6 avr. 2018 20:21, "Kenneth Knowles" < >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> >> >>>> <mailto:[email protected]>> a écrit : >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> I'm working on finding a solution for >>>>>>>>>>>>>> launching the Nexmark >>>>>>>>>>>>>> >> >>>> suite with each runner. This doesn't have to >>>>>>>>>>>>>> be done via >>>>>>>>>>>>>> >> >>>> Gradle, >>>>>>>>>>>>>> >> >>>> but we anyhow need built artifacts that >>>>>>>>>>>>>> don't require user >>>>>>>>>>>>>> >> >>>> classpath intervention. >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> It looks to me like the examples are also >>>>>>>>>>>>>> missing this - >>>>>>>>>>>>>> >> >>>> they >>>>>>>>>>>>>> >> >>>> have separate configuration e.g. >>>>>>>>>>>>>> sparkRunnerPreCommit but >>>>>>>>>>>>>> >> >>>> that >>>>>>>>>>>>>> >> >>>> is overspecified compared to a free-form >>>>>>>>>>>>>> launching of a >>>>>>>>>>>>>> >> >>>> main() >>>>>>>>>>>>>> >> >>>> program with a runner profile. >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik >>>>>>>>>>>>>> >> >>>> <[email protected] >>>>>>>>>>>>>> >> >>>> <mailto:[email protected]>> wrote: >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> Romain, are you talking about the >>>>>>>>>>>>>> profiles that exist as >>>>>>>>>>>>>> >> >>>> part of the archetype examples? >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> If so, then those still exist and >>>>>>>>>>>>>> haven't been changed. >>>>>>>>>>>>>> >> >>>> If >>>>>>>>>>>>>> >> >>>> not, can you provide a link to the >>>>>>>>>>>>>> profile in a pom file >>>>>>>>>>>>>> >> >>>> to >>>>>>>>>>>>>> >> >>>> be clearer? >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> On Fri, Apr 6, 2018 at 12:40 PM Romain >>>>>>>>>>>>>> Manni-Bucau >>>>>>>>>>>>>> >> >>>> <[email protected] <mailto: >>>>>>>>>>>>>> [email protected]>> >>>>>>>>>>>>>> >> >>>> wrote: >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> Hi Scott, >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> is it right that 2 doesn't handle >>>>>>>>>>>>>> the hierachy >>>>>>>>>>>>>> >> >>>> anymore >>>>>>>>>>>>>> >> >>>> and that it doesn't handle profiles >>>>>>>>>>>>>> for runners as >>>>>>>>>>>>>> >> >>>> it is >>>>>>>>>>>>>> >> >>>> currently with maven? >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> Romain Manni-Bucau >>>>>>>>>>>>>> >> >>>> @rmannibucau < >>>>>>>>>>>>>> https://twitter.com/rmannibucau> | >>>>>>>>>>>>>> >> >>>> Blog >>>>>>>>>>>>>> >> >>>> <https://rmannibucau.metawerx.net/> >>>>>>>>>>>>>> | Old Blog >>>>>>>>>>>>>> >> >>>> <http://rmannibucau.wordpress.com> >>>>>>>>>>>>>> | Github >>>>>>>>>>>>>> >> >>>> <https://github.com/rmannibucau> | >>>>>>>>>>>>>> LinkedIn >>>>>>>>>>>>>> >> >>>> < >>>>>>>>>>>>>> https://www.linkedin.com/in/rmannibucau> | Book >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> < >>>>>>>>>>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> 2018-04-06 18:32 GMT+02:00 Scott >>>>>>>>>>>>>> Wegner >>>>>>>>>>>>>> >> >>>> <[email protected] <mailto: >>>>>>>>>>>>>> [email protected]>>: >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> I wanted to start a thread to >>>>>>>>>>>>>> summarize the >>>>>>>>>>>>>> >> >>>> current >>>>>>>>>>>>>> >> >>>> state of Gradle migration. We've >>>>>>>>>>>>>> made lots of >>>>>>>>>>>>>> >> >>>> good >>>>>>>>>>>>>> >> >>>> progress so far this week. >>>>>>>>>>>>>> Here's the status >>>>>>>>>>>>>> >> >>>> from >>>>>>>>>>>>>> >> >>>> what I can tell-- please add or >>>>>>>>>>>>>> correct anything >>>>>>>>>>>>>> >> >>>> I >>>>>>>>>>>>>> >> >>>> missed: >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> * Release artifacts can be built >>>>>>>>>>>>>> and published >>>>>>>>>>>>>> >> >>>> for >>>>>>>>>>>>>> >> >>>> Snapshot and officlal releases >>>>>>>>>>>>>> [1] >>>>>>>>>>>>>> >> >>>> * Gradle-generated releases have >>>>>>>>>>>>>> been validated >>>>>>>>>>>>>> >> >>>> with >>>>>>>>>>>>>> >> >>>> the the Apache Beam archetype >>>>>>>>>>>>>> generation >>>>>>>>>>>>>> >> >>>> quickstart; >>>>>>>>>>>>>> >> >>>> still needs additional >>>>>>>>>>>>>> validation. >>>>>>>>>>>>>> >> >>>> * Generated release pom files >>>>>>>>>>>>>> have correct >>>>>>>>>>>>>> >> >>>> project >>>>>>>>>>>>>> >> >>>> metadata [2] >>>>>>>>>>>>>> >> >>>> * The python pre-commits are now >>>>>>>>>>>>>> working in >>>>>>>>>>>>>> >> >>>> Gradle >>>>>>>>>>>>>> >> >>>> [3] >>>>>>>>>>>>>> >> >>>> * Ismaël has started a >>>>>>>>>>>>>> collaborative doc of >>>>>>>>>>>>>> >> >>>> Gradle >>>>>>>>>>>>>> >> >>>> tips [4] as we all learn the new >>>>>>>>>>>>>> system-- please >>>>>>>>>>>>>> >> >>>> add >>>>>>>>>>>>>> >> >>>> your own. This will eventually >>>>>>>>>>>>>> feed into >>>>>>>>>>>>>> >> >>>> official >>>>>>>>>>>>>> >> >>>> documentation on the website. >>>>>>>>>>>>>> >> >>>> * Łukasz Gajowy is working on >>>>>>>>>>>>>> migrating >>>>>>>>>>>>>> >> >>>> performance >>>>>>>>>>>>>> >> >>>> testing framework [5] >>>>>>>>>>>>>> >> >>>> * Daniel is working on updating >>>>>>>>>>>>>> documentation to >>>>>>>>>>>>>> >> >>>> refer to Gradle instead of maven >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> If I missed anything, please add >>>>>>>>>>>>>> it to this >>>>>>>>>>>>>> >> >>>> thread. >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> The general roadmap we're >>>>>>>>>>>>>> working towards is: >>>>>>>>>>>>>> >> >>>> (a) Publish release artifacts >>>>>>>>>>>>>> with Gradle >>>>>>>>>>>>>> >> >>>> (SNAPSHOT >>>>>>>>>>>>>> >> >>>> and signed releases) >>>>>>>>>>>>>> >> >>>> (b) Postcommits migrated to >>>>>>>>>>>>>> Gradle >>>>>>>>>>>>>> >> >>>> (c) Migrate documentation from >>>>>>>>>>>>>> maven to Gradle >>>>>>>>>>>>>> >> >>>> (d) Migrate perfkit suites to >>>>>>>>>>>>>> use Gradle >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> For those of you that are >>>>>>>>>>>>>> hacking: thanks for >>>>>>>>>>>>>> >> >>>> your >>>>>>>>>>>>>> >> >>>> help so far! Progress is being >>>>>>>>>>>>>> roughly tracked >>>>>>>>>>>>>> >> >>>> on >>>>>>>>>>>>>> >> >>>> the Kanban [6]; please make sure >>>>>>>>>>>>>> the issues >>>>>>>>>>>>>> >> >>>> assigned >>>>>>>>>>>>>> >> >>>> to you are up-to-date. Many of >>>>>>>>>>>>>> the changes are >>>>>>>>>>>>>> >> >>>> staged on lukecwik's local >>>>>>>>>>>>>> branch [7]; we'll >>>>>>>>>>>>>> >> >>>> work on >>>>>>>>>>>>>> >> >>>> merging them back soon. >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> [1] >>>>>>>>>>>>>> >> >>>> https://github.com/lukecwik/incubator-beam/pull/7 >>>>>>>>>>>>>> >> >>>> [2] >>>>>>>>>>>>>> >> >>>> https://github.com/lukecwik/incubator-beam/pull/3 >>>>>>>>>>>>>> >> >>>> [3] >>>>>>>>>>>>>> https://github.com/apache/beam/pull/5032 >>>>>>>>>>>>>> >> >>>> [4] >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit >>>>>>>>>>>>>> >> >>>> [5] >>>>>>>>>>>>>> https://github.com/apache/beam/pull/5003 >>>>>>>>>>>>>> >> >>>> [6] >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242 >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> [7] >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> https://github.com/lukecwik/incubator-beam/tree/gradle >>>>>>>>>>>>>> >> >>>> -- >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> Got feedback? >>>>>>>>>>>>>> http://go/swegner-feedback >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>
