Alexey, are you referring to tests run with "./gradlew :beam-runners-direct-java:needsRunnerTests"? That command works fine for me in both versions of IDEA, but I believe the same tests fail if you run them directly through "./gradlew test".
However, I am having issues with a bunch of validatesRunner tests, mostly be caused by :beam-runners-google-cloud-dataflow-java:validatesRunner. Not sure if it's because of a code change or a gradle config, I'll keep looking into it. On Wed, Apr 11, 2018 at 11:01 AM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > I got tests running rrconfiguring gradle (which was setup for another > project but seems beam didnt like it) but latency is still "high" using > gradle runner for tests (like Etienne said ~3s on an i7 with 16G vs a few > ms with default idea test runner, would be great to solve that). > > I also find the integration quite fishy cause configurations are customs > so idea is kind of forced to propose your modukes 3 times at least when you > select the classpath (x_test being generally the working one). > > Also the false positive you get if you forget a cleanX is a bit annoying, > maybe we should force a clean for test or at least when there is a --tests > to avoid gradle to not run it cause there is no diff. > > So it works but dev productivity is reduced a lot and it became slow > enough to think if you should do a contribution or not - at least for me. > > Le 11 avr. 2018 19:37, "Alexey Romanenko" <aromanenko....@gmail.com> a > écrit : > >> I’ve managed to import a project as it’s described in documentation >> (starting from empty project) using Idea 2018 and run unit tests >> successfully. >> For some reasons, tests, that use DirectRunner to run a pipeline, were >> failed. >> >> WBR, >> Alexey >> >> On 11 Apr 2018, at 19:01, Daniel Oliveira <danolive...@google.com> wrote: >> >> Hi everyone, I was the one who initially wrote the PR with Idea >> instructions <https://github.com/apache/beam-site/pull/414>. I was using >> 2017.3 as well while writing it so all the instructions were tested on that >> version. I'll try testing the instructions on 2018 to see if I can >> reproduce the issues people are having. >> >> On Wed, Apr 11, 2018 at 9:51 AM Lukasz Cwik <lc...@google.com> wrote: >> >>> I use 2017.3 and it has been reliable for me. I haven't tried 2018 yet. >>> >>> On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau < >>> rmannibu...@gmail.com> wrote: >>> >>>> Any of you using the idea 2018? the import works for me but then it is >>>> not as smooth as it seems for you. I'm just trying to see if it is a >>>> procedure thing or a version issue. >>>> >>>> Romain Manni-Bucau >>>> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>> >>>> >>>> 2018-04-11 17:28 GMT+02:00 Kenneth Knowles <k...@google.com>: >>>> > The only reason I did "empty project then add a module" procedure was >>>> to get >>>> > all the IntelliJ files outside the source tree. IIRC directly >>>> creating from >>>> > existing sources didn't give the necessary configuration options. If >>>> you >>>> > don't care about being able to `git clean` then you can do the shorter >>>> > version. Also the particular UI for creation might have improved >>>> since I >>>> > tried it. I'll do it again. >>>> > >>>> > On the pom, since it is only generated for machine reading, why care >>>> if the >>>> > parent is inlined? I actually prefer to avoid coupling with things >>>> that you >>>> > have to go and look up. Using inheritance is OK for human edited poms >>>> > (actually IMO it is still a mistake) but it doesn't change the >>>> semantics of >>>> > a shipped pom if they are all immutable, which they should be. It is >>>> better >>>> > to have all the info right there. Is there an actually effective >>>> difference? >>>> > >>>> > Kenn >>>> > >>>> > On Wed, Apr 11, 2018 at 6:53 AM Etienne Chauchot < >>>> echauc...@apache.org> >>>> > wrote: >>>> >> >>>> >> Hi all, >>>> >> I just tested gradle environment from a fresh source clone with this >>>> >> procedure with just a tiny change: I used "new project from existing >>>> >> sources" rather than create empty project and then add module. >>>> >> >>>> >> It works fine and junit runs from intellij also work. with gradle >>>> we pay >>>> >> a 2s delay (on my machine) before running the actual test for each >>>> run. This >>>> >> delay seems quite constant no matter the module: I have run all the >>>> tests at >>>> >> once in runner-core and a single test in another module with a >>>> similar >>>> >> delay. >>>> >> >>>> >> I also tested a debug session from intellij and it runs fine also. >>>> >> >>>> >> I'll do some more testing and keep you posted. >>>> >> >>>> >> I still have some concerns about the potential difficulty for new >>>> >> contributors to have to learn gradle in addition to Beam itself >>>> comparing to >>>> >> maven which is more mainstream for java developers. That could be >>>> >> discouraging for ex for part-time contributors >>>> >> >>>> >> Etienne >>>> >> >>>> >> Le mardi 10 avril 2018 à 14:38 +0000, Lukasz Cwik a écrit : >>>> >> >>>> >> beam-site PR/414 updates the instructions for using Intellij and how >>>> to >>>> >> import a module: >>>> >> >>>> >> 1. Create an empty IntelliJ project outside of the Beam source tree. >>>> >> 2. Under Project Structure > Project, select a Project SDK. >>>> >> 3. Under Project Structure > Modules, click the + sign to add a >>>> module and >>>> >> select "Import Module". >>>> >> 1. Select the directory containing the Beam source tree. >>>> >> 2. Tick the "Import module from external model" button and select >>>> >> Gradle >>>> >> from the list. >>>> >> 3. Tick the following boxes. >>>> >> * Use auto-import >>>> >> * Create separate module per source set >>>> >> * Store generated project files externally >>>> >> * Use default gradle wrapper >>>> >> 4. Delegate build actions to Gradle by going to Settings > Build, >>>> >> Execution, >>>> >> Deployment > Build Tools > Gradle and checking "Delegate IDE >>>> build/run >>>> >> actions to gradle". >>>> >> >>>> >> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré < >>>> j...@nanthrax.net> >>>> >> wrote: >>>> >> >>>> >> That's a very important issue for contribution. Up to now, I used >>>> Maven >>>> >> for setup IntelliJ (and it works just fine). If we remove the >>>> pom.xml, >>>> >> we have to support Eclipse and IntelliJ "smoothly". >>>> >> >>>> >> Let me try in IntelliJ. >>>> >> >>>> >> Regards >>>> >> JB >>>> >> >>>> >> On 10/04/2018 15:21, Romain Manni-Bucau wrote: >>>> >> > You dont have issue due to the build setup with that option. I get: >>>> >> > >>>> >> > avr. 10, 2018 3:20:10 PM >>>> >> > org.apache.beam.runners.direct.DirectTransformExecutor run >>>> >> > GRAVE: Error occurred within >>>> >> > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a >>>> >> > com.google.common.util.concurrent.ExecutionError: >>>> >> > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy >>>> >> > >>>> >> > ? >>>> >> > >>>> >> > Romain Manni-Bucau >>>> >> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>> >> > >>>> >> > >>>> >> > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik <lc...@google.com>: >>>> >> >> I have found that the simplest setup is to delegate the build/test >>>> >> >> actions >>>> >> >> to Gradle. This allows you to run unit tests very easily and >>>> since its >>>> >> >> in >>>> >> >> the same manner that Gradle would have, you know that if its >>>> passing it >>>> >> >> will >>>> >> >> pass on the command line and on Jenkins. Here is one site that >>>> >> >> discusses how >>>> >> >> to set this up: >>>> >> >> >>>> >> >> >>>> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html >>>> >> >> >>>> >> >> >>>> >> >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau >>>> >> >> <rmannibu...@gmail.com> >>>> >> >> wrote: >>>> >> >>> >>>> >> >>> What's the plan to make idea supporting gradle on beam project? >>>> Do we >>>> >> >>> import the workaround mentionned in >>>> >> >>> https://youtrack.jetbrains.com/issue/IDEA-175172? >>>> >> >>> For the ones who didn't see this issue in action: idea will >>>> compile in >>>> >> >>> out/ instead of build/ and you will just miss all the resources >>>> you >>>> >> >>> need like some SPI registration which are used by all our >>>> registrar => >>>> >> >>> no way to run tests in idea without hacking the configuration >>>> quite >>>> >> >>> deeply :( >>>> >> >>> >>>> >> >>> Romain Manni-Bucau >>>> >> >>> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>> >> >>> >>>> >> >>> >>>> >> >>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot < >>>> echauc...@apache.org>: >>>> >> >>>> As a gradle beginner, I could not agree more ! >>>> >> >>>> +1 >>>> >> >>>> Etienne >>>> >> >>>> Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a >>>> écrit : >>>> >> >>>> >>>> >> >>>> Hi all, >>>> >> >>>> >>>> >> >>>> I did multiple gradle build since last week and I would like to >>>> share >>>> >> >>>> one of my concern: it's about the communities. >>>> >> >>>> >>>> >> >>>> If I think our users won't see any change for them due to Gradle >>>> >> >>>> build >>>> >> >>>> (I think that most of our users will still use Maven with >>>> artifacts >>>> >> >>>> provided by Gradle), I'm more concerned by the dev community >>>> and the >>>> >> >>>> contribution. >>>> >> >>>> >>>> >> >>>> Maven is well known and straight forward for a large part of >>>> >> >>>> potential >>>> >> >>>> contributors. I think we have to keep in mind that we still >>>> have to >>>> >> >>>> grow >>>> >> >>>> up our contributors community. >>>> >> >>>> >>>> >> >>>> Today, maybe I'm wrong, but I have the feeling that gradle >>>> build is >>>> >> >>>> not >>>> >> >>>> straight forward (build.gradle includes build_rules.gradle, >>>> gathering >>>> >> >>>> all taks all together). >>>> >> >>>> >>>> >> >>>> I would like to add a task in the gradle "migration" process: >>>> >> >>>> simplify >>>> >> >>>> the gradle structure and files, and document this. >>>> >> >>>> >>>> >> >>>> I know we already have a Jira about the documentation part, but >>>> I >>>> >> >>>> would >>>> >> >>>> like to "polish" and use a clean structure for the Gradle >>>> resources. >>>> >> >>>> As >>>> >> >>>> already quickly discussed, I think that having one gradle file >>>> per >>>> >> >>>> tasks >>>> >> >>>> in the .gradle directory would be helpful. >>>> >> >>>> >>>> >> >>>> The goal is really to simplify the contribution. >>>> >> >>>> >>>> >> >>>> Do you agree if I add a Jira about "Gradle polish" ? >>>> >> >>>> Thoughts ? >>>> >> >>>> >>>> >> >>>> Regards >>>> >> >>>> JB >>>> >> >>>> >>>> >> >>>> On 07/04/2018 04:52, Scott Wegner wrote: >>>> >> >>>> >>>> >> >>>> Here's an end-of-day update on migration work: >>>> >> >>>> >>>> >> >>>> * Snapshot unsigned dailies and signed release builds are >>>> working >>>> >> >>>> (!!). >>>> >> >>>> PR/5048 [1] merges changes from Luke's branch >>>> >> >>>> * python precommit failing... will investigate python >>>> precommit >>>> >> >>>> Monday >>>> >> >>>> * All Precommits are gradle only >>>> >> >>>> * All Postcommits except performance tests and >>>> Java_JDK_Versions_Test >>>> >> >>>> use gradle (after PR/5047 [2] merged) >>>> >> >>>> * Nightly snapshot release using gradle is ready; needs PR/5048 >>>> to be >>>> >> >>>> merged before switching >>>> >> >>>> * ValidatesRunner_Spark failing consistently; investigating >>>> >> >>>> >>>> >> >>>> Thanks for another productive day of hacking. I'll pick up >>>> again on >>>> >> >>>> Monday. >>>> >> >>>> >>>> >> >>>> [1] https://github.com/apache/beam/pull/5048 >>>> >> >>>> [2] https://github.com/apache/beam/pull/5047 >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau >>>> >> >>>> <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> wrote: >>>> >> >>>> >>>> >> >>>> Why building a zip per runner which its stack and just >>>> pointing >>>> >> >>>> out >>>> >> >>>> on that zip and let beam lazy load the runner: >>>> >> >>>> >>>> >> >>>> --runner=LazyRunner --lazyRunnerDir=... >>>> --lazyRunnerOptions=... >>>> >> >>>> (or >>>> >> >>>> the fromSystemProperties() if it gets merged a day ;)) >>>> >> >>>> >>>> >> >>>> Le 6 avr. 2018 20:21, "Kenneth Knowles" <k...@google.com >>>> >> >>>> <mailto:k...@google.com>> a écrit : >>>> >> >>>> >>>> >> >>>> I'm working on finding a solution for launching the >>>> Nexmark >>>> >> >>>> suite with each runner. This doesn't have to be done >>>> via >>>> >> >>>> Gradle, >>>> >> >>>> but we anyhow need built artifacts that don't require >>>> user >>>> >> >>>> classpath intervention. >>>> >> >>>> >>>> >> >>>> It looks to me like the examples are also missing this >>>> - >>>> >> >>>> they >>>> >> >>>> have separate configuration e.g. sparkRunnerPreCommit >>>> but >>>> >> >>>> that >>>> >> >>>> is overspecified compared to a free-form launching of a >>>> >> >>>> main() >>>> >> >>>> program with a runner profile. >>>> >> >>>> >>>> >> >>>> On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik >>>> >> >>>> <lc...@google.com >>>> >> >>>> <mailto:lc...@google.com>> wrote: >>>> >> >>>> >>>> >> >>>> Romain, are you talking about the profiles that >>>> exist as >>>> >> >>>> part of the archetype examples? >>>> >> >>>> >>>> >> >>>> If so, then those still exist and haven't been >>>> changed. >>>> >> >>>> If >>>> >> >>>> not, can you provide a link to the profile in a >>>> pom file >>>> >> >>>> to >>>> >> >>>> be clearer? >>>> >> >>>> >>>> >> >>>> On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau >>>> >> >>>> <rmannibu...@gmail.com <mailto: >>>> rmannibu...@gmail.com>> >>>> >> >>>> wrote: >>>> >> >>>> >>>> >> >>>> Hi Scott, >>>> >> >>>> >>>> >> >>>> is it right that 2 doesn't handle the hierachy >>>> >> >>>> anymore >>>> >> >>>> and that it doesn't handle profiles for >>>> runners as >>>> >> >>>> it is >>>> >> >>>> currently with maven? >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> Romain Manni-Bucau >>>> >> >>>> @rmannibucau <https://twitter.com/rmannibucau> >>>> | >>>> >> >>>> Blog >>>> >> >>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>> >> >>>> <http://rmannibucau.wordpress.com> | Github >>>> >> >>>> <https://github.com/rmannibucau> | LinkedIn >>>> >> >>>> <https://www.linkedin.com/in/rmannibucau> | >>>> Book >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> < >>>> https://www.packtpub.com/application-development/java-ee-8-high-performance >>>> > >>>> >> >>>> >>>> >> >>>> 2018-04-06 18:32 GMT+02:00 Scott Wegner >>>> >> >>>> <sweg...@google.com <mailto:sweg...@google.com >>>> >>: >>>> >> >>>> >>>> >> >>>> I wanted to start a thread to summarize the >>>> >> >>>> current >>>> >> >>>> state of Gradle migration. We've made lots >>>> of >>>> >> >>>> good >>>> >> >>>> progress so far this week. Here's the >>>> status >>>> >> >>>> from >>>> >> >>>> what I can tell-- please add or correct >>>> anything >>>> >> >>>> I >>>> >> >>>> missed: >>>> >> >>>> >>>> >> >>>> * Release artifacts can be built and >>>> published >>>> >> >>>> for >>>> >> >>>> Snapshot and officlal releases [1] >>>> >> >>>> * Gradle-generated releases have been >>>> validated >>>> >> >>>> with >>>> >> >>>> the the Apache Beam archetype generation >>>> >> >>>> quickstart; >>>> >> >>>> still needs additional validation. >>>> >> >>>> * Generated release pom files have correct >>>> >> >>>> project >>>> >> >>>> metadata [2] >>>> >> >>>> * The python pre-commits are now working in >>>> >> >>>> Gradle >>>> >> >>>> [3] >>>> >> >>>> * Ismaël has started a collaborative doc of >>>> >> >>>> Gradle >>>> >> >>>> tips [4] as we all learn the new system-- >>>> please >>>> >> >>>> add >>>> >> >>>> your own. This will eventually feed into >>>> >> >>>> official >>>> >> >>>> documentation on the website. >>>> >> >>>> * Łukasz Gajowy is working on migrating >>>> >> >>>> performance >>>> >> >>>> testing framework [5] >>>> >> >>>> * Daniel is working on updating >>>> documentation to >>>> >> >>>> refer to Gradle instead of maven >>>> >> >>>> >>>> >> >>>> If I missed anything, please add it to this >>>> >> >>>> thread. >>>> >> >>>> >>>> >> >>>> The general roadmap we're working towards >>>> is: >>>> >> >>>> (a) Publish release artifacts with Gradle >>>> >> >>>> (SNAPSHOT >>>> >> >>>> and signed releases) >>>> >> >>>> (b) Postcommits migrated to Gradle >>>> >> >>>> (c) Migrate documentation from maven to >>>> Gradle >>>> >> >>>> (d) Migrate perfkit suites to use Gradle >>>> >> >>>> >>>> >> >>>> For those of you that are hacking: thanks >>>> for >>>> >> >>>> your >>>> >> >>>> help so far! Progress is being roughly >>>> tracked >>>> >> >>>> on >>>> >> >>>> the Kanban [6]; please make sure the issues >>>> >> >>>> assigned >>>> >> >>>> to you are up-to-date. Many of the changes >>>> are >>>> >> >>>> staged on lukecwik's local branch [7]; >>>> we'll >>>> >> >>>> work on >>>> >> >>>> merging them back soon. >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> [1] >>>> >> >>>> https://github.com/lukecwik/incubator-beam/pull/7 >>>> >> >>>> [2] >>>> >> >>>> https://github.com/lukecwik/incubator-beam/pull/3 >>>> >> >>>> [3] >>>> https://github.com/apache/beam/pull/5032 >>>> >> >>>> [4] >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> >>>> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit >>>> >> >>>> [5] >>>> https://github.com/apache/beam/pull/5003 >>>> >> >>>> [6] >>>> >> >>>> >>>> >> >>>> >>>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242 >>>> >> >>>> >>>> >> >>>> [7] >>>> >> >>>> >>>> >> >>>> https://github.com/lukecwik/incubator-beam/tree/gradle >>>> >> >>>> -- >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> Got feedback? http://go/swegner-feedback >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> > >>>> >>> >>
