Re: Gradle Status [April 6]

2018-04-11 Thread Lukasz Cwik
Romain, it seems like you want us to generate an exact copy of the existing poms that we have. There is a lot of stuff in there related to building/testing the Apache Beam project which our users don't need (integration/test profiles, test scope dependencies, parents, ...). Users who want to build

Development of new HTTP source

2018-04-11 Thread Daniel Salerno
Good morning, In my Big Data Google project we need to read batch data from the VTEX platform every 15 minutes and record json's return to our cloud storage datalake. It makes the data available through HTTP GET requests for its API: (

Re: Development of new HTTP source

2018-04-11 Thread Romain Manni-Bucau
Hi Daniel, I know JB has a started a RestIO, it is not yet complete and needs some love but probably some opportunity for converge and collaboration here Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book 2018-04-11 15:42 GMT+02:00 Daniel Salerno

Re: Gradle Status [April 6]

2018-04-11 Thread Etienne Chauchot
Hi all, I just tested gradle environment from a fresh source clone with this procedure with just a tiny change: I used "new project from existing sources" rather than create empty project and then add module.  It works fine and junit runs from intellij also work.  with gradle we pay a 2s delay

Re: Gradle Status [April 6]

2018-04-11 Thread Romain Manni-Bucau
Hi guys, checked the snapshot pom today and there are issues: 1. no parent (so 2 is hard to handle, no properties etc) 2. too much noise informations (all the parent data should be in the parent) 3. wrong scopes (all is compile) - which is likely a leak of gradle scripts 4. missing important

Re: Gradle Status [April 6]

2018-04-11 Thread Kenneth Knowles
The only reason I did "empty project then add a module" procedure was to get all the IntelliJ files outside the source tree. IIRC directly creating from existing sources didn't give the necessary configuration options. If you don't care about being able to `git clean` then you can do the shorter

Re: Gradle Status [April 6]

2018-04-11 Thread Romain Manni-Bucau
Any of you using the idea 2018? the import works for me but then it is not as smooth as it seems for you. I'm just trying to see if it is a procedure thing or a version issue. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book 2018-04-11 17:28 GMT+02:00 Kenneth

Re: Development of new HTTP source

2018-04-11 Thread Ismaël Mejía
There is also Romain's filesystem PR that wraps vfs as a Beam filesystem . That would at least in theory allow to read from HTTP and FTP, no? Of course this is different from RestIO because it won't have the full HTTP verbs semantics but if the goal is just to read from (GET), maybe we should try

Re: org.apache.beam.sdk.values.TupleTag#genId and stacktraces?

2018-04-11 Thread Reuven Lax
I think it's best for the users to pick their own identifier for TupleTags. As part of the snapshot/update proposal, we want to be able update pipelines across refactorings. We tell people to supply a stable name for transforms (in apply) specifically to make this doable (so we can match up

Re: Gradle Status [April 6]

2018-04-11 Thread Lukasz Cwik
I use 2017.3 and it has been reliable for me. I haven't tried 2018 yet. On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau wrote: > Any of you using the idea 2018? the import works for me but then it is > not as smooth as it seems for you. I'm just trying to see if it is

Re: Running Nexmark with PubSub issue

2018-04-11 Thread Raghu Angadi
I noticed it too while adding KafkaIO support for Nexmark (this was in parallel to another PR for KafkaIO that got merged recently). The anonymous inner class for DoFn is not serializable. I moved it to a static class in my branch, but didn't test it yet :

Re: Gradle Status [April 6]

2018-04-11 Thread Romain Manni-Bucau
I got tests running rrconfiguring gradle (which was setup for another project but seems beam didnt like it) but latency is still "high" using gradle runner for tests (like Etienne said ~3s on an i7 with 16G vs a few ms with default idea test runner, would be great to solve that). I also find the

Build failed in Jenkins: beam_SeedJob #1480

2018-04-11 Thread Apache Jenkins Server
See -- GitHub pull request #5101 of commit 0a27a08743d6547867df89f7cc76a343107f46d0, no merge conflicts. Setting status of 0a27a08743d6547867df89f7cc76a343107f46d0 to PENDING with url

Re: Build failed in Jenkins: beam_SeedJob #1480

2018-04-11 Thread Alan Myrvold
Typo in my pull request. Fixing it now. On Wed, Apr 11, 2018 at 11:42 AM Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See > > -- > GitHub pull request #5101 of commit >

Jenkins build is back to normal : beam_SeedJob #1481

2018-04-11 Thread Apache Jenkins Server
See

Re: Gradle Status [April 11]

2018-04-11 Thread Kenneth Knowles
Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner + Netty has been introduced since yesterday. Etienne mentioned he has worked toward setting up periodic runs on all runners, so this should help get us towards that. We'll probably prefer to build standalone fat jars for

Gradle Status [April 11]

2018-04-11 Thread Scott Wegner
Thanks everyone for the continued effort towards the Gradle migration. As a high-level summary of our progress since Friday: we have a viable build, with a number of minor issues that we're still working out. Please take a look at the new documentation in our contribution guide and log any bugs

Re: Gradle Status [April 6]

2018-04-11 Thread Alexey Romanenko
I’ve managed to import a project as it’s described in documentation (starting from empty project) using Idea 2018 and run unit tests successfully. For some reasons, tests, that use DirectRunner to run a pipeline, were failed. WBR, Alexey > On 11 Apr 2018, at 19:01, Daniel Oliveira

Re: Development of new HTTP source

2018-04-11 Thread Romain Manni-Bucau
I guess for now you can feth his fork or just refork it and pr on it directly. Le 11 avr. 2018 19:21, "Daniel Salerno" a écrit : > Hello Romain, > > Thanks for the feedback. > It sounds like a good idea! > How could I get the rest IO from JB? > > Thank you. > > 2018-04-11

Re: Gradle Status [April 6]

2018-04-11 Thread Daniel Oliveira
Alexey, are you referring to tests run with "./gradlew :beam-runners-direct-java:needsRunnerTests"? That command works fine for me in both versions of IDEA, but I believe the same tests fail if you run them directly through "./gradlew test". However, I am having issues with a bunch of

Re: Gradle Status [April 6]

2018-04-11 Thread Daniel Oliveira
Hi everyone, I was the one who initially wrote the PR with Idea instructions . I was using 2017.3 as well while writing it so all the instructions were tested on that version. I'll try testing the instructions on 2018 to see if I can reproduce the

Re: Development of new HTTP source

2018-04-11 Thread Daniel Salerno
Hello Romain, Thanks for the feedback. It sounds like a good idea! How could I get the rest IO from JB? Thank you. 2018-04-11 10:50 GMT-03:00 Romain Manni-Bucau : > Hi Daniel, > > I know JB has a started a RestIO, it is not yet complete and needs > some love but probably

Re: Permission to get JIRAs assigned to me

2018-04-11 Thread Boyuan Zhang
Hey Kenn, Thanks so much for your help! Boyuan Zhang On Wed, Apr 11, 2018 at 3:54 PM Kenneth Knowles wrote: > You need to be given the "Contributors" role in JIRA. I have done so - > give it a try. > > Kenn > > On Wed, Apr 11, 2018 at 3:24 PM Boyuan Zhang

Re: Gradle Status [April 11]

2018-04-11 Thread Nathan Fisher
Is there a document or similar outlining the decision behind Gradle over Bazel? For purely intellectual curiosity I’m curious what tradeoffs/benefits were considered when evaluating the two or if it was more a matter of community/contributor familiarity. I found this thread which seems to imply

Re: Permission to get JIRAs assigned to me

2018-04-11 Thread Kenneth Knowles
You need to be given the "Contributors" role in JIRA. I have done so - give it a try. Kenn On Wed, Apr 11, 2018 at 3:24 PM Boyuan Zhang wrote: > Hey all, > > I created several JIRAs in beam and wanna assign them to myself but it > looks like I don't have right permissions

Permission to get JIRAs assigned to me

2018-04-11 Thread Boyuan Zhang
Hey all, I created several JIRAs in beam and wanna assign them to myself but it looks like I don't have right permissions yet. Could you please help me to figure out how to get right permissions? By the way, I have filed ICLA. Thanks so much for all your help! Boyuan Zhang

Re: Gradle Status [April 11]

2018-04-11 Thread Kenneth Knowles
There are plenty of technical hurdles since Bazel is best suited to an isolated monorepo. We could probably have overcome those problems with effort. I got through a few of them in the short sprint I did. Ultimately it was a community-driven decision: more people (as in "more than just me" :-)

Re: [PROPOSAL] Preparing 2.5.0 release next week

2018-04-11 Thread Ismaël Mejía
+1 to delay 2 weeks as Ahmet proposes. We can cut the branch in two weeks in a more stable shape doing it right now is not a good idea. We can justify the delay on the impact of transitioning the build system. On Wed, Apr 11, 2018 at 10:35 PM, Ahmet Altay wrote: > +1 to

Re: [PROPOSAL] Preparing 2.5.0 release next week

2018-04-11 Thread Romain Manni-Bucau
Any hope the release is on central before the 30th? Le 11 avr. 2018 22:02, "Robert Bradshaw" a écrit : > +1 to keeping up the regular release cycle, but I don't think it makes > sense to cut until we're ready to actively work on the release. A cut date > two weeks from now

Re: [PROPOSAL] Preparing 2.5.0 release next week

2018-04-11 Thread Chamikara Jayalath
Hi JB, Please note that I opened a blocker [1] (working on it) and looks like we have several other JIRAs marked for 2.5.0. So +1 for waiting for two weeks (or till JIRAs are resolved or moved off the list). Thanks, Cham [1] https://issues.apache.org/jira/browse/BEAM-3991 [2]

Re: [PROPOSAL] Preparing 2.5.0 release next week

2018-04-11 Thread Robert Bradshaw
+1 to keeping up the regular release cycle, but I don't think it makes sense to cut until we're ready to actively work on the release. A cut date two weeks from now seems fine (unless someone else is volunteering to do it this time around). That being said, a dry run using gradle now may make a

Re: Development of new HTTP source

2018-04-11 Thread Daniel Salerno
Sorry Romain, I did not get it right ... Are you saying so for now, develop from scratch the source Http? Sorry I'm new to the group, what does PR mean? 2018-04-11 14:38 GMT-03:00 Romain Manni-Bucau : > I guess for now you can feth his fork or just refork it and pr on it

Re: Gradle Status [April 6]

2018-04-11 Thread Lukasz Cwik
Running Dataflow validates runner tests requires you to have permission to launch jobs on the 'apache-beam-testing' project. You'll need to override dataflowProject and dataflowTempRoot with a GCP project and GCS bucket you have access to. On Wed, Apr 11, 2018 at 3:17 PM Daniel Oliveira

Re: [PROPOSAL] Preparing 2.5.0 release next week

2018-04-11 Thread Jean-Baptiste Onofré
Hi guys, Due to the last work, I think it makes sense to try a release using Gradle. If it doesn't work smoothly, we will rollback to Maven, and maybe in that case, we should ask ourselves if Gradle is really a good alternative for now. I'm in vacation tomorrow morning for 2 weeks. I can

Re: Gradle questions on Eclipse and End to End tests

2018-04-11 Thread Scott Wegner
FYI, I was able to get basic Gradle integration working in Eclipse; take a look at documentation here: https://github.com/apache/beam-site/pull/419 I'm not a regular Eclipse user, so there may be other pieces that I'm not aware of. But the Gradle integration seems quite mature. Feel free to add

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

2018-04-11 Thread Alex Amato
Thank you everyone for your feedback so far. I have made a revision today which is to make all metrics refer to a primary entity, so I have restructured some of the protos a little bit. The point of this change was to futureproof the possibility of allowing custom user metrics, with custom

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

2018-04-11 Thread Ben Chambers
The metric api is designed to prevent user defined metric types based on the fact they just weren't used enough to justify support. Is there a reason we are bringing that complexity back? Shouldn't we just need the ability for the standard set plus any special system metrivs? On Wed, Apr 11,

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

2018-04-11 Thread Robert Bradshaw
Fully custom metric types is the "more speculative and difficult" feature that I was proposing we kick down the road (and may never get to). What I'm suggesting is that we support custom metrics of standard type. On Wed, Apr 11, 2018 at 5:52 PM Ben Chambers wrote: > The

Re: [PROPOSAL] Preparing 2.5.0 release next week

2018-04-11 Thread Jean-Baptiste Onofré
+1 agree to wait two weeks. So I will tackle the release when back from vacation. Thanks guys ! I can take off serene ;) Regards JB On 04/11/2018 11:20 PM, Ismaël Mejía wrote: > +1 to delay 2 weeks as Ahmet proposes. We can cut the branch in two > weeks in a more stable shape doing it right

Re: [PROPOSAL] Preparing 2.5.0 release next week

2018-04-11 Thread Romain Manni-Bucau
Is it possible to delay it of 2 more weeks then? ~16th? Without a lot of effort on pom generation, maven consumer will be broken and I will ot be able to work on that (or something else ;)) the two first weeks of may. Alternatively, a compromise can be to push existing poms until we drop them.

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

2018-04-11 Thread Ben Chambers
When you say type do you mean accumulator type, result type, or accumulator strategy? Specifically, what is the "type" of sumint, sumlong, meanlong, etc? On Wed, Apr 11, 2018, 9:38 PM Robert Bradshaw wrote: > Fully custom metric types is the "more speculative and difficult"