Re: org.apache.beam.sdk.values.TupleTag#genId and stacktraces?

Le 10 avr. 2018 22:59, "Robert Bradshaw" a écrit : On Tue, Apr 10, 2018 at 1:49 PM Romain Manni-Bucau wrote: > > Le 10 avr. 2018 21:25, "Robert Bradshaw" a écrit : > > On Tue, Apr 10, 2018 at 12:10 PM Romain Manni-Bucau > wrote: > >> This is interesting cause it leads to "why do the workers n

Re: Gradle Status [April 6]

Le 11 avr. 2018 02:30, "Reuven Lax" a écrit : Actually I always found the right-click to run tests to only sometimes work in Maven, especially if there were changes to dependent AutoValue classes where code had to be generated. Too often it would fail, and I would then need to use Maven to rebuil

Re: Gradle Status [April 6]

Actually I always found the right-click to run tests to only sometimes work in Maven, especially if there were changes to dependent AutoValue classes where code had to be generated. Too often it would fail, and I would then need to use Maven to rebuild the whole project. It would be cool if Gradle

Re: Python postcommit and precommit

I think we should replace the shell script with a top level pythonPostCommit gradle target, similar to the precomment. On Mon, Apr 9, 2018 at 12:12 PM Lukasz Cwik wrote: > The shell scripts still exist instead of using Gradle. Migrating to Gradle > as the build system hasn't addressed this (only

Re: Gradle Status [April 6]

Reuven's point is good. Once we hit the bare minimum of having things working, let's collect usability improvements and engineering improvements on a separate JIRA from the main migration. I filed https://issues.apache.org/jira/browse/BEAM-4045 for these less critical issues to separate them from

Jenkins build is back to normal : beam_SeedJob #1471

See

Build failed in Jenkins: beam_SeedJob #1470

See -- GitHub pull request #5088 of commit a5ebf9146e9e05427de5b28f8900578a4cec8949, no merge conflicts. Setting status of a5ebf9146e9e05427de5b28f8900578a4cec8949 to PENDING with url https

Build failed in Jenkins: beam_SeedJob #1469

See -- GitHub pull request #5088 of commit 56ba4f1bde57937c7a1c2d30f7588847a6489d30, no merge conflicts. Setting status of 56ba4f1bde57937c7a1c2d30f7588847a6489d30 to PENDING with url https

Re: org.apache.beam.sdk.values.TupleTag#genId and stacktraces?

On Tue, Apr 10, 2018 at 1:49 PM Romain Manni-Bucau wrote: > > Le 10 avr. 2018 21:25, "Robert Bradshaw" a écrit : > > On Tue, Apr 10, 2018 at 12:10 PM Romain Manni-Bucau > wrote: > >> This is interesting cause it leads to "why do the workers need to do it >> again instead of reusing the computed

Re: org.apache.beam.sdk.values.TupleTag#genId and stacktraces?

Le 10 avr. 2018 21:25, "Robert Bradshaw" a écrit : On Tue, Apr 10, 2018 at 12:10 PM Romain Manni-Bucau wrote: > This is interesting cause it leads to "why do the workers need to do it > again instead of reusing the computed one?". Technically the answer is > trivial but in terms of design I thi

Re: Gradle Status [April 6]

@jb: what did you change? I re-imported the project like 3 times earlier today and never got it working acceptably :( Personally if importing the project and right click on a test+debug works as good as maven in idea id be happy. I can manage other stuff in a console even if gradle reporting is no

Re: Gradle Status [April 6]

There are a lot of ideas on how to increase usability, but I think they'll get lost in the thread. I suggest we try to capture them in Jiras. I suggest we also find out what common use patterns are (people on this thread are probably sufficient), as different people will have different workflows.

Re: org.apache.beam.sdk.values.TupleTag#genId and stacktraces?

On Tue, Apr 10, 2018 at 12:10 PM Romain Manni-Bucau wrote: > This is interesting cause it leads to "why do the workers need to do it > again instead of reusing the computed one?". Technically the answer is > trivial but in terms of design I think beam tends to abuse static init > block - even in

Re: Gradle Status [April 6]

FYI, I did a new attempt and it works fine (pretty long). Previous try failed. Regards JB On 10/04/2018 19:52, Kenneth Knowles wrote: I've been on Idea+Gradle for ~two months, around the time I added https://github.com/apache/beam/pull/4583 and https://github.com/apache/beam/pull/4626 to make

Re: Gradle Status [April 6]

Le 10 avr. 2018 19:53, "Kenneth Knowles" a écrit : I've been on Idea+Gradle for ~two months, around the time I added https://github.com/apache/beam/pull/4583 and https://github.com/apache/ beam/pull/4626 to make the import require zero user work. I have no fear of deleting my project any time and

Re: org.apache.beam.sdk.values.TupleTag#genId and stacktraces?

This is interesting cause it leads to "why do the workers need to do it again instead of reusing the computed one?". Technically the answer is trivial but in terms of design I think beam tends to abuse static init block - even in dofn api - which easily lead to issues when we will want to support m

Re: org.apache.beam.sdk.values.TupleTag#genId and stacktraces?

I believe it doesn't need to be stable across refactoring, only across all workers executing a specific version of the code. Specifically, it is used as follows: 1. Create a pipeline on the user's machine. It walks the stack until the static initializer block, which provides an ID. 2. Send the pip

Re: Gradle Status [April 6]

I've been on Idea+Gradle for ~two months, around the time I added https://github.com/apache/beam/pull/4583 and https://github.com/apache/beam/pull/4626 to make the import require zero user work. I have no fear of deleting my project any time and re-importing. I agree with not having auto-import on

Re: org.apache.beam.sdk.values.TupleTag#genId and stacktraces?

Well issue is more about all the existing tests currently. Out of curiosity: how walking the stack is stable since the stack can change? Stop condition is the static block of a class which can use method so refactoring and therefore is not stable. Should it be deprecated? Le 10 avr. 2018 19:17, "

Re: org.apache.beam.sdk.values.TupleTag#genId and stacktraces?

If it's too slow perhaps you could use the constructor where you pass an explicit id (though in my experience walking the stack isn't that slow). On Tue, Apr 10, 2018 at 10:09 AM Romain Manni-Bucau wrote: > Oops cross post sorry. > > Issue i hit on this thread is it is used a lot in tests abd it

Re: org.apache.beam.sdk.values.TupleTag#genId and stacktraces?

It may be reasonable to port most of those TupleTags to have an explicit, rather than generated ID, which will remove the need to inspect the stack trace. However, as mentioned, the constructor shouldn't provide an unstable ID, as otherwise most pipelines won't work on production runners. On Tue,

Re: org.apache.beam.sdk.values.TupleTag#genId and stacktraces?

Le 10 avr. 2018 18:40, "Robert Bradshaw" a écrit : These values should be, inasmuch as possible, stable across VMs. How slow is slow? Doesn't this happen only once per VM startup? Once per jvm and idea launches a jvm per test and the daemon does save enough time, you still go through the whole

Re: org.apache.beam.sdk.values.TupleTag#genId and stacktraces?

Oops cross post sorry. Issue i hit on this thread is it is used a lot in tests abd it slows down tests for nothing like with generatesequence ones Le 10 avr. 2018 19:00, "Romain Manni-Bucau" a écrit : > > > Le 10 avr. 2018 18:40, "Robert Bradshaw" a écrit : > > These values should be, inasmuch

Re: org.apache.beam.sdk.values.TupleTag#genId and stacktraces?

In fact, this is explicitly to work with `static final` TupleTags, and using a non-stable isn't feasible. A static final TupleTag won't be serialized in the closure of an object that uses it - it will be instantiated independently in any other ClassLoader, such as on a remote JVM. If you use a con

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

I've gathered a lot of feedback so far and want to make a decision by Friday, and begin working on related PRs next week. Please make sure that you provide your feedback before then and I will post the final decisions made to this thread Friday afternoon. On Thu, Apr 5, 2018 at 1:38 AM Ismaël Mej

Re: org.apache.beam.sdk.values.TupleTag#genId and stacktraces?

These values should be, inasmuch as possible, stable across VMs. How slow is slow? Doesn't this happen only once per VM startup? On Tue, Apr 10, 2018 at 9:33 AM Romain Manni-Bucau wrote: > Hi > > does org.apache.beam.sdk.values.TupleTag#genId need to get the > stacktrace or can we use any id gen

org.apache.beam.sdk.values.TupleTag#genId and stacktraces?

Hi does org.apache.beam.sdk.values.TupleTag#genId need to get the stacktrace or can we use any id generator (like UUID.random().toString())? Using traces is quite slow under load and environments where the root stack is not just the "next" level so skipping it would be nice. Romain Manni-Bucau @r

Re: Gradle Status [April 6]

Runner a test doesnt have the right classpath (idea uses out/ instead of build/) then when you switch on gradle runner the launching uses gradle which is not able to use submodules directly but reconsider the whole project which is quite slow for normal dev iterations compare to just run the test w

Re: Gradle Status [April 6]

Romain, Can you detail what's not working. I switched my IntelliJ over to Gradle about two weeks ago, and haven't had any trouble. Reuven On Tue, Apr 10, 2018 at 4:20 PM Romain Manni-Bucau wrote: > Ok, didn't find a way to make it working properly (only workaround > with direct commands and no

Re: Gradle Status [April 6]

Ok, didn't find a way to make it working properly (only workaround with direct commands and no good idea integration for debugging). I'm back with maven, if anyone knows how to properly solve it let's do it. If not I think JB point is to consider more than any other criteria. Romain Manni-Bucau @r

Re: Gradle Status [April 6]

side note: do NOT use auto-import until you are sure you can, it locks regularly on beam (pby too big for idea?) and makes idea ready to be killed :( Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book 2018-04-10 16:40 GMT+02:00 Jean-Baptiste Onofré : > It's what I did,

Re: Gradle Status [April 6]

It's what I did, I'm trying a complete reload now (maybe this step failed). On 10/04/2018 16:38, Lukasz Cwik wrote: beam-site PR/414 updates the instructions for using Intellij and how to import a module: 1. Create an empty IntelliJ project outside of the Beam source tree. 2. Under Project Str

Re: Gradle Status [April 6]

beam-site PR/414 updates the instructions for using Intellij and how to import a module: 1. Create an empty IntelliJ project outside of the Beam source tree. 2. Under Project Structure > Project, select a Project SDK. 3. Under Project Structure > Modules, click the + sign to add a module and se

Re: Gradle Status [April 6]

That's a very important issue for contribution. Up to now, I used Maven for setup IntelliJ (and it works just fine). If we remove the pom.xml, we have to support Eclipse and IntelliJ "smoothly". Let me try in IntelliJ. Regards JB On 10/04/2018 15:21, Romain Manni-Bucau wrote: You dont have i

Re: Gradle Status [April 6]

Romain, I haven't seen that error. At the very top of your test execution log it gives you the tasks that it is running, for example: 6:41:33 AM: Executing tasks ':beam-sdks-java-core:cleanTest :beam-sdks-java-core:test --tests "org.apache.beam.sdk.coders.AvroCoderTest.testAvroCoderEncoding"'... W

Re: [PROPOSAL] Preparing 2.5.0 release next week

Nightly snapshots have migrated to being produced via Gradle. Alan Myrvold and a few others have been working on adding automated tests that validate those nightly snapshots by running through the quickstarts available on the website. Please try out the nightly snapshot and report bugs as sub-task

Re: Gradle Status [April 6]

You dont have issue due to the build setup with that option. I get: avr. 10, 2018 3:20:10 PM org.apache.beam.runners.direct.DirectTransformExecutor run GRAVE: Error occurred within org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a com.google.common.util.concurrent.ExecutionError: jav

Re: Gradle Status [April 6]

I have found that the simplest setup is to delegate the build/test actions to Gradle. This allows you to run unit tests very easily and since its in the same manner that Gradle would have, you know that if its passing it will pass on the command line and on Jenkins. Here is one site that discusses

Re: Gradle Status [April 6]

What's the plan to make idea supporting gradle on beam project? Do we import the workaround mentionned in https://youtrack.jetbrains.com/issue/IDEA-175172? For the ones who didn't see this issue in action: idea will compile in out/ instead of build/ and you will just miss all the resources you need

Re: [PROPOSAL] Preparing 2.5.0 release next week

+1000 to Romain's point on dependencies, we have to obsessively pay attention to the consistency of the dependencies, this is critical for users and we cannot radically change the produced artifacts or we risk of breaking their applications.. On Tue, Apr 10, 2018 at 6:56 AM, Romain Manni-Bucau w

Re: DirectRunner in test - await completion of workers threads?

It seems there is still an issue with teardown not being called in failed tasks, just created BEAM-4040 to track it. On Thu, Apr 5, 2018 at 4:45 PM, Tim Robertson wrote: > Will do - I'll report the result on https://github.com/apache/beam/pull/4905 > > On Thu, Apr 5, 2018 at 11:45 AM, Ismaël Mejí

Re: Gradle Status [April 6]

As a gradle beginner, I could not agree more !  +1 Etienne Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a écrit : > Hi all, > > I did multiple gradle build since last week and I would like to share  > one of my concern: it's about the communities. > > If I think our users won't see