Re: Gradle Status [April 6]

Caching success/failure of a test is legitimate useful behavior. That said, I'm not sure how it interacts with Idea. Kenn On Sun, Apr 15, 2018 at 12:23 AM Romain Manni-Bucau wrote: > If you dont the diff is empty and gradle runs nothing, no? Saw it with > gradle 4.6 > >

Re: Gradle Status [April 6]

If you dont the diff is empty and gradle runs nothing, no? Saw it with gradle 4.6 Le 15 avr. 2018 00:49, "Kenneth Knowles" a écrit : > You shouldn't do :module:cleanTest. If that is necessary that's a major > bug in the build. > > Kenn > > On Fri, Apr 13, 2018 at 11:46 PM

Re: Gradle Status [April 6]

You shouldn't do :module:cleanTest. If that is necessary that's a major bug in the build. Kenn On Fri, Apr 13, 2018 at 11:46 PM Romain Manni-Bucau wrote: > There is a fake module xxx_test which should have the right classpath but > since idea compilation is messed up you

Re: Gradle Status [April 6]

There is a fake module xxx_test which should have the right classpath but since idea compilation is messed up you will still have to run :module:cleanTest :module:test --tests org...MyTest.myMethod, even with idea which leads to the same latency as the command line :( Le 13 avr. 2018 22:23,

Re: Gradle Status [April 6]

Is there a Jira for this 3 second delay? Also you're initial complaint was not about the 3 second delay, so it wasn't clear that's what you were complaining about. Reuven On Fri, Apr 13, 2018 at 4:42 AM Romain Manni-Bucau wrote: > When you launch a test with gradle

Re: Gradle Status [April 6]

Ah, I see. I was attempting to replicate Alexey's issues with DirectRunner tests, so I wasn't checking for that 3s overhead. A quick test for me shows that I also get that several second overhead when running tests with Gradle, ranging from 3-6 sec. However, on my machine this delay is constant,

Re: Gradle Status [April 6]

I also don't quite understand what your question is, and it appears like Dan spent considerable time trying to reproduce your issue. For the record, I have had no issues running tests via Gradle in IntelliJ for the past few weeks. Reuven On Thu, Apr 12, 2018 at 9:47 PM Daniel Oliveira

Re: Gradle Status [April 6]

Sorry Romain, I'm not quite sure what you're asking. Can you clarify? On Thu, Apr 12, 2018 at 12:22 PM Romain Manni-Bucau wrote: > Well you are the only one to not have the drawbacks to use it so maybe > dont do it? I know Luke is in holidays but anyone else with the

Re: Gradle Status [April 6]

Well you are the only one to not have the drawbacks to use it so maybe dont do it? I know Luke is in holidays but anyone else with the knowledge of why we nees that noise compared to idea native tooling/flow? Le 12 avr. 2018 20:16, "Daniel Oliveira" a écrit : > Ah, I did

Re: Gradle Status [April 6]

Ah, I did not. Thanks Romain. I tried it again, restarting in between, and still had no differences. Since it seems like there's no reason not to use "Gradle Test Runner", I'll mention it in the contributor's guide. On Thu, Apr 12, 2018 at 10:31 AM Romain Manni-Bucau

Re: Gradle Status [April 6]

@Daniel: did you restart in between? Otherwise it does nothing. One launches JunitCoreRunner from idea and the other a gradle command. Le 12 avr. 2018 19:24, "Daniel Oliveira" a écrit : > I think it depends on what exactly switching to "Gradle Test Runner" from >

Re: Gradle Status [April 6]

I think it depends on what exactly switching to "Gradle Test Runner" from "Platform Test Runner" does. I tried it out on my machine and they seem to act identically to each other. The IntelliJ documentation says it determines what API to use to run the tests

Re: Gradle Status [April 6]

Daniel, actually I did run it with default IDEA JUnit test runner. Then, in “Settings > Build, Execution, Deployment > Build Tools > Gradle > Runner" I selected “Gradle Test Runner” in “Run tests using” selectbox and it works ok when I run my tests with IDEA shortcuts. So, probably, we should

Re: Gradle Status [April 6]

My test was also on version 2017, 2017.2.7 to be precise Etienne Le mercredi 11 avril 2018 à 17:01 +, Daniel Oliveira a écrit : > Hi everyone, I was the one who initially wrote the PR with Idea instructions. > I was using 2017.3 as well while writing > it so all the instructions were tested

Re: Gradle Status [April 6]

Running Dataflow validates runner tests requires you to have permission to launch jobs on the 'apache-beam-testing' project. You'll need to override dataflowProject and dataflowTempRoot with a GCP project and GCS bucket you have access to. On Wed, Apr 11, 2018 at 3:17 PM Daniel Oliveira

Re: Gradle Status [April 6]

Alexey, are you referring to tests run with "./gradlew :beam-runners-direct-java:needsRunnerTests"? That command works fine for me in both versions of IDEA, but I believe the same tests fail if you run them directly through "./gradlew test". However, I am having issues with a bunch of

Re: Gradle Status [April 6]

I got tests running rrconfiguring gradle (which was setup for another project but seems beam didnt like it) but latency is still "high" using gradle runner for tests (like Etienne said ~3s on an i7 with 16G vs a few ms with default idea test runner, would be great to solve that). I also find the

Re: Gradle Status [April 6]

I’ve managed to import a project as it’s described in documentation (starting from empty project) using Idea 2018 and run unit tests successfully. For some reasons, tests, that use DirectRunner to run a pipeline, were failed. WBR, Alexey > On 11 Apr 2018, at 19:01, Daniel Oliveira

Re: Gradle Status [April 6]

Hi everyone, I was the one who initially wrote the PR with Idea instructions . I was using 2017.3 as well while writing it so all the instructions were tested on that version. I'll try testing the instructions on 2018 to see if I can reproduce the

Re: Gradle Status [April 6]

I use 2017.3 and it has been reliable for me. I haven't tried 2018 yet. On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau wrote: > Any of you using the idea 2018? the import works for me but then it is > not as smooth as it seems for you. I'm just trying to see if it is

Re: Gradle Status [April 6]

Any of you using the idea 2018? the import works for me but then it is not as smooth as it seems for you. I'm just trying to see if it is a procedure thing or a version issue. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book 2018-04-11 17:28 GMT+02:00 Kenneth

Re: Gradle Status [April 6]

The only reason I did "empty project then add a module" procedure was to get all the IntelliJ files outside the source tree. IIRC directly creating from existing sources didn't give the necessary configuration options. If you don't care about being able to `git clean` then you can do the shorter

Re: Gradle Status [April 6]

Hi all, I just tested gradle environment from a fresh source clone with this procedure with just a tiny change: I used "new project from existing sources" rather than create empty project and then add module.  It works fine and junit runs from intellij also work.  with gradle we pay a 2s delay

Re: Gradle Status [April 6]

Romain, it seems like you want us to generate an exact copy of the existing poms that we have. There is a lot of stuff in there related to building/testing the Apache Beam project which our users don't need (integration/test profiles, test scope dependencies, parents, ...). Users who want to build

Re: Gradle Status [April 6]

Hi guys, checked the snapshot pom today and there are issues: 1. no parent (so 2 is hard to handle, no properties etc) 2. too much noise informations (all the parent data should be in the parent) 3. wrong scopes (all is compile) - which is likely a leak of gradle scripts 4. missing important

Re: Gradle Status [April 6]

Le 11 avr. 2018 02:30, "Reuven Lax" a écrit : Actually I always found the right-click to run tests to only sometimes work in Maven, especially if there were changes to dependent AutoValue classes where code had to be generated. Too often it would fail, and I would then need to

Re: Gradle Status [April 6]

Actually I always found the right-click to run tests to only sometimes work in Maven, especially if there were changes to dependent AutoValue classes where code had to be generated. Too often it would fail, and I would then need to use Maven to rebuild the whole project. It would be cool if Gradle

Re: Gradle Status [April 6]

Reuven's point is good. Once we hit the bare minimum of having things working, let's collect usability improvements and engineering improvements on a separate JIRA from the main migration. I filed https://issues.apache.org/jira/browse/BEAM-4045 for these less critical issues to separate them

Re: Gradle Status [April 6]

@jb: what did you change? I re-imported the project like 3 times earlier today and never got it working acceptably :( Personally if importing the project and right click on a test+debug works as good as maven in idea id be happy. I can manage other stuff in a console even if gradle reporting is

Re: Gradle Status [April 6]

There are a lot of ideas on how to increase usability, but I think they'll get lost in the thread. I suggest we try to capture them in Jiras. I suggest we also find out what common use patterns are (people on this thread are probably sufficient), as different people will have different workflows.

Re: Gradle Status [April 6]

FYI, I did a new attempt and it works fine (pretty long). Previous try failed. Regards JB On 10/04/2018 19:52, Kenneth Knowles wrote: I've been on Idea+Gradle for ~two months, around the time I added https://github.com/apache/beam/pull/4583 and https://github.com/apache/beam/pull/4626 to

Re: Gradle Status [April 6]

Le 10 avr. 2018 19:53, "Kenneth Knowles" a écrit : I've been on Idea+Gradle for ~two months, around the time I added https://github.com/apache/beam/pull/4583 and https://github.com/apache/ beam/pull/4626 to make the import require zero user work. I have no fear of deleting my

Re: Gradle Status [April 6]

I've been on Idea+Gradle for ~two months, around the time I added https://github.com/apache/beam/pull/4583 and https://github.com/apache/beam/pull/4626 to make the import require zero user work. I have no fear of deleting my project any time and re-importing. I agree with not having auto-import

Re: Gradle Status [April 6]

Runner a test doesnt have the right classpath (idea uses out/ instead of build/) then when you switch on gradle runner the launching uses gradle which is not able to use submodules directly but reconsider the whole project which is quite slow for normal dev iterations compare to just run the test

Re: Gradle Status [April 6]

Romain, Can you detail what's not working. I switched my IntelliJ over to Gradle about two weeks ago, and haven't had any trouble. Reuven On Tue, Apr 10, 2018 at 4:20 PM Romain Manni-Bucau wrote: > Ok, didn't find a way to make it working properly (only workaround >

Re: Gradle Status [April 6]

Ok, didn't find a way to make it working properly (only workaround with direct commands and no good idea integration for debugging). I'm back with maven, if anyone knows how to properly solve it let's do it. If not I think JB point is to consider more than any other criteria. Romain Manni-Bucau

Re: Gradle Status [April 6]

side note: do NOT use auto-import until you are sure you can, it locks regularly on beam (pby too big for idea?) and makes idea ready to be killed :( Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book 2018-04-10 16:40 GMT+02:00 Jean-Baptiste Onofré

Re: Gradle Status [April 6]

It's what I did, I'm trying a complete reload now (maybe this step failed). On 10/04/2018 16:38, Lukasz Cwik wrote: beam-site PR/414 updates the instructions for using Intellij and how to import a module: 1. Create an empty IntelliJ project outside of the Beam source tree. 2. Under Project

Re: Gradle Status [April 6]

beam-site PR/414 updates the instructions for using Intellij and how to import a module: 1. Create an empty IntelliJ project outside of the Beam source tree. 2. Under Project Structure > Project, select a Project SDK. 3. Under Project Structure > Modules, click the + sign to add a module and

Re: Gradle Status [April 6]

That's a very important issue for contribution. Up to now, I used Maven for setup IntelliJ (and it works just fine). If we remove the pom.xml, we have to support Eclipse and IntelliJ "smoothly". Let me try in IntelliJ. Regards JB On 10/04/2018 15:21, Romain Manni-Bucau wrote: You dont have

Re: Gradle Status [April 6]

Romain, I haven't seen that error. At the very top of your test execution log it gives you the tasks that it is running, for example: 6:41:33 AM: Executing tasks ':beam-sdks-java-core:cleanTest :beam-sdks-java-core:test --tests "org.apache.beam.sdk.coders.AvroCoderTest.testAvroCoderEncoding"'...

Re: Gradle Status [April 6]

You dont have issue due to the build setup with that option. I get: avr. 10, 2018 3:20:10 PM org.apache.beam.runners.direct.DirectTransformExecutor run GRAVE: Error occurred within org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a com.google.common.util.concurrent.ExecutionError:

Re: Gradle Status [April 6]

I have found that the simplest setup is to delegate the build/test actions to Gradle. This allows you to run unit tests very easily and since its in the same manner that Gradle would have, you know that if its passing it will pass on the command line and on Jenkins. Here is one site that discusses

Re: Gradle Status [April 6]

What's the plan to make idea supporting gradle on beam project? Do we import the workaround mentionned in https://youtrack.jetbrains.com/issue/IDEA-175172? For the ones who didn't see this issue in action: idea will compile in out/ instead of build/ and you will just miss all the resources you

Re: Gradle Status [April 6]

As a gradle beginner, I could not agree more !  +1 Etienne Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a écrit : > Hi all, > > I did multiple gradle build since last week and I would like to share  > one of my concern: it's about the communities. > > If I think our users won't see

Re: Gradle Status [April 6]

Simplifying examples/java precommit is encompassed in BEAM-4033. Both you and Kenn pointed to the same thing. To my knowledge it is the only place where we have such a loop that generates tasks. On Mon, Apr 9, 2018 at 3:37 PM Romain Manni-Bucau wrote: > > > Le 9 avr. 2018

Re: Gradle Status [April 6]

Le 9 avr. 2018 21:04, "Lukasz Cwik" a écrit : Romain, can you clarify by the weird task naming? (Give some examples using our current project and Gradle and what you would have expected.) Sure

Re: Gradle Status [April 6]

Hi Luke, let me take a concrete example. I'm developing a Beam extension. Most of the time, I will ask myself: 1. How do I do the build.gradle and include my extension in Beam project 2. How do my extension is compile (the equivalent of maven-compiler-plugin) and how to deal with dependency

Re: Gradle Status [April 6]

Romain, can you clarify by the weird task naming? (Give some examples using our current project and Gradle and what you would have expected.) On Mon, Apr 9, 2018 at 2:49 PM Romain Manni-Bucau wrote: > +1 gradle not being mainstream - and even more in beam scope than other

Re: Gradle Status [April 6]

+1 gradle not being mainstream - and even more in beam scope than other java scope - it must stay simple. I know that personally it takes me x2 or 3 to say "ok ill fork" a project using gradle, in particular with custom scripts and not only parent/module scripts. If needed we can write custom

Re: Gradle Status [April 6]

My phrasing sounded a bit too much like adding a blocking condition. I absolutely don't want to do that. Our maven build was far from clear, and had lots of tech debt and spooky action at a distance, I just want to emphatically agree with JB's sentiment that we have an opportunity to improve it

Re: Gradle Status [April 6]

Fixed up prior e-mail. On Mon, Apr 9, 2018 at 1:50 PM Lukasz Cwik wrote: > JB, learning the build system in a project is hopefully avoided by most > users if the contribution guide can clearly explain what users need to do. > But for everyone else who wants to change a

Re: Gradle Status [April 6]

JB, learning the build system in a project is hopefully avoided by most users if the contribution guide can clearly explain what users need to do. But for everyone else who wants to change a dependency version or add a dependency it should be as simple as copy/paste (which I believe it already

Re: Gradle Status [April 6]

Huge +1 to the idea of investing in simplification and polish of the Gradle files before considering the migration complete. 1. build.gradle files should be as close to straight-line configuration as possible: - You should be able to understand a module's build easily, locally, without knowing

Re: Gradle Status [April 6]

Hi all, I did multiple gradle build since last week and I would like to share one of my concern: it's about the communities. If I think our users won't see any change for them due to Gradle build (I think that most of our users will still use Maven with artifacts provided by Gradle), I'm

Re: Gradle Status [April 6]

On Sat, Apr 7, 2018 at 12:43 Reuven Lax wrote: > So if I understand correctly, we've migrated all precommit, most > postcommits, and we have a working release process using Gradle. There are > a few bugs left, but at this pace it sounds like we're close to fully > migrated. > >

Re: Gradle Status [April 6]

So if I understand correctly, we've migrated all precommit, most postcommits, and we have a working release process using Gradle. There are a few bugs left, but at this pace it sounds like we're close to fully migrated. I know that multiple people put it long hours last getting this done last

Re: Gradle Status [April 6]

Here's an end-of-day update on migration work: * Snapshot unsigned dailies and signed release builds are working (!!). PR/5048 [1] merges changes from Luke's branch * python precommit failing... will investigate python precommit Monday * All Precommits are gradle only * All Postcommits except

Re: Gradle Status [April 6]

Le 6 avr. 2018 20:09, "Lukasz Cwik" a écrit : Romain, are you talking about the profiles that exist as part of the archetype examples? Was more thinking to this kind of profiles https://github.com/apache/beam/blob/master/sdks/java/nexmark/pom.xml (which should hit all IO at

Re: Gradle Status [April 6]

I'm working on finding a solution for launching the Nexmark suite with each runner. This doesn't have to be done via Gradle, but we anyhow need built artifacts that don't require user classpath intervention. It looks to me like the examples are also missing this - they have separate configuration

Re: Gradle Status [April 6]

Romain, are you talking about the profiles that exist as part of the archetype examples? If so, then those still exist and haven't been changed. If not, can you provide a link to the profile in a pom file to be clearer? On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau

Gradle Status [April 6]

I wanted to start a thread to summarize the current state of Gradle migration. We've made lots of good progress so far this week. Here's the status from what I can tell-- please add or correct anything I missed: * Release artifacts can be built and published for Snapshot and officlal releases [1]